同步不同的 Sidekiq 线程的方法并等待

Question

>问题：我有几个sidekiq线程和一个函数，一次只能从任何线程调用一次。

原因：我们正在查询 AdWords API 以获取一些数据。在速率限制方面，它们非常严格。一次只能有一个线程调用该函数来获取数据。

现在一些代码：

# Public: Get estimates for a set of keywords. If there is an error, retry
# several times. If not successful, raise an error
#
# keywords: The keyword objects to get estimates for.
# save: Boolean to indicate whether the keyword objects should be saved to
# the database
#
def repeatedly_try_get_estimates(keywords: [], save: true, sleep_delay: 150)
return keywords if keywords.empty?
func = -> { get_estimates(keywords, !save) }
retry_operation(function: func, max_tries: 15, sleep_delay: sleep_delay)
end

• 如您所见，现在我有一个巨大的sleep_delay来解决这个问题。
• 代码retry_operation使用get_estimates函数作为参数。然后，它将重试 多次get_estimates函数，直到有 API 例外。

retry_function：

# Private: Retry a function X times and wait X seconds. If it does not work X times,
# raise an error. If successful return the functions results.
#
# - max_tries: The maximum tries to repeat the function
# - sleep_delay: The seconds to wait between each iteration.
# - function: The lambda function to call each iteration
#
def retry_operation(max_tries: 5, sleep_delay: 30, function: nil, current_try: 0, result: nil)
# Can't call, no function
if function.nil?
return
end
# Abort, tried too frequently.
if current_try > max_tries
raise "Failed function too often"
end
# Check if there is an exception
exception = true
begin
result = function.call
exception = false
rescue => e
Rails.logger.info "Received error when repeatedly calling function #{e.message.to_s}"
end
if exception
sleep sleep_delay if sleep_delay > 0
retry_operation(max_tries: max_tries, sleep_delay: sleep_delay, function: function, current_try: current_try + 1)
else
result
end
end

get_estimates_function在这里：https://gist.github.com/a14868d939ef0e34ef9f。它太长了，以防万一。

我想我需要执行以下操作：

1. 调整repeatedly_try_get_estimates函数中的代码。
2. 在类中使用互斥锁。
3. 如果互斥锁正在使用中，则挽救异常。
4. 只有当互斥锁空闲时，运行rety_operation，否则休眠一段时间

感谢您的帮助:)

Answer 1

我们开始了，让它工作：

# Public: Get estimates for a set of keywords. If there is an error, retry
# several times. If not successful, raise an error
#
# keywords: The keyword objects to get estimates for.
# save: Boolean to indicate whether the keyword objects should be saved to
# the database
#
def repeatedly_try_get_estimates(keywords: [], save: true, sleep_delay: 40)
return keywords if keywords.empty?
func = -> { get_estimates(keywords, save_keywords: true) }
exception = nil
result = nil
initial_sleep = 0
estimates_mutex.synchronize do
since_last_request = Time.now.to_i - last_adwords_api_request
if since_last_request <= 30
Rails.logger.info "AdWords: Last request was only few seconds ago - sleeping #{since_last_request}."
initial_sleep = since_last_request
end
begin
result = retry_operation(function: func, max_tries: 15, sleep_delay: sleep_delay, initial_sleep: initial_sleep)
rescue => e
exception = e
end
@@last_adwords_api_request = Time.now.to_i
end
if exception
raise exception
end
result
end