在python中矢量化循环

Question

我有一个以下循环，我正在计算不同大小的批次softmax变换

如下

import numpy as np 
def softmax(Z,arr):
"""
:param Z:  numpy array of any shape (output from hidden layer)
:param arr: numpy array of any shape (start, end)
:return A: output of multinum_logit(Z,arr), same shape as Z
:return cache: returns Z as well, useful during back propagation
"""
A = np.zeros(Z.shape)
for i in prange(len(arr)):
shiftx = Z[:,arr[i,1]:arr[i,2]+1] - np.max(Z[:,int(arr[i,1]):int(arr[i,2])+1])
A[:,arr[i,1]:arr[i,2]+1] = np.exp(shiftx)/np.exp(shiftx).sum()
cache = Z
return A,cache

由于这个 for 循环没有矢量化，因此它是我代码中的瓶颈。什么是使其更快的可能解决方案。我尝试使用@jitnumba，这使得它更快一点，但还不够。我想知道是否有另一种方法可以使其更快或矢量化/并行化它。

函数的示例输入数据

Z = np.random.random([1,10000])
arr = np.zeros([100,3])
arr[:,0] = 1
temp = int(Z.shape[1]/arr.shape[0])
for i in range(arr.shape[0]):
arr[i,1] = i*temp
arr[i,2] = (i+1)*temp-1
arr = arr.astype(int)

编辑：

我忘了在这里强调我的班级数量是不同的。例如，第 1 批有 10 个类，第 2 批可能有 15 个类。因此，我正在传递一个数组 arr，该数组跟踪哪些行属于 batch1 等。这些批次与传统神经网络框架中的批次不同

在上面的示例中，arr跟踪行的起始索引和结束索引。因此，softmax 函数中的分母将仅是索引位于起始索引和结束索引之间的那些观测值的总和。

Answer 1

这是一个矢量化的softmax函数。这是斯坦福大学关于卷积网络的cs231n课程作业的实现。

该函数接受可优化的参数、输入数据、目标和正则化器。(您可以忽略正则化器，因为它引用了某些 cs231n 赋值独有的另一个类)。

它返回参数的损失和梯度。

def softmax_loss_vectorized(W, X, y, reg):
"""
Softmax loss function, vectorized version.
Inputs and outputs are the same as softmax_loss_naive.
"""
# Initialize the loss and gradient to zero.
loss = 0.0
dW = np.zeros_like(W)
num_train = X.shape[0]
scores = X.dot(W)
shift_scores = scores - np.amax(scores,axis=1).reshape(-1,1)
softmax = np.exp(shift_scores)/np.sum(np.exp(shift_scores), axis=1).reshape(-1,1)
loss = -np.sum(np.log(softmax[range(num_train), list(y)]))
loss /= num_train
loss += 0.5* reg * np.sum(W * W)
dSoftmax = softmax.copy()
dSoftmax[range(num_train), list(y)] += -1
dW = (X.T).dot(dSoftmax)
dW = dW/num_train + reg * W
return loss, dW

为了比较起见，这里是同一方法的朴素(非矢量化)实现。

def softmax_loss_naive(W, X, y, reg):
"""
Softmax loss function, naive implementation (with loops)
Inputs have dimension D, there are C classes, and we operate on minibatches
of N examples.
Inputs:
- W: A numpy array of shape (D, C) containing weights.
- X: A numpy array of shape (N, D) containing a minibatch of data.
- y: A numpy array of shape (N,) containing training labels; y[i] = c means
that X[i] has label c, where 0 <= c < C.
- reg: (float) regularization strength
Returns a tuple of:
- loss as single float
- gradient with respect to weights W; an array of same shape as W
"""
loss = 0.0
dW = np.zeros_like(W)
num_train = X.shape[0]
num_classes = W.shape[1]
for i in xrange(num_train):
scores = X[i].dot(W)
shift_scores = scores - max(scores)
loss_i = -shift_scores[y[i]] + np.log(sum(np.exp(shift_scores)))
loss += loss_i
for j in xrange(num_classes):
softmax = np.exp(shift_scores[j])/sum(np.exp(shift_scores))
if j==y[i]:
dW[:,j] += (-1 + softmax) * X[i]
else:
dW[:,j] += softmax *X[i]
loss /= num_train
loss += 0.5 * reg * np.sum(W * W)
dW /= num_train + reg * W
return loss, dW

源