otx.getall() having error "Exceeded maximum number of retries" #55

Psr18delhi · 2020-08-30T08:25:27Z

MaxRetryError Traceback (most recent call last)
C:\ProgramData\Anaconda3\lib\site-packages\requests\adapters.py in send(self, request, stream, timeout, verify, cert, proxies)
448 retries=self.max_retries,
--> 449 timeout=timeout
450 )

C:\ProgramData\Anaconda3\lib\site-packages\urllib3\connectionpool.py in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
730 release_conn=release_conn,
--> 731 body_pos=body_pos, **response_kw)
732

C:\ProgramData\Anaconda3\lib\site-packages\urllib3\connectionpool.py in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
710 try:
--> 711 retries = retries.increment(method, url, response=response, _pool=self)
712 except MaxRetryError:

C:\ProgramData\Anaconda3\lib\site-packages\urllib3\util\retry.py in increment(self, method, url, response, error, _pool, _stacktrace)
398 if new_retry.is_exhausted():
--> 399 raise MaxRetryError(_pool, url, error or ResponseError(cause))
400

MaxRetryError: HTTPSConnectionPool(host='otx.alienvault.com', port=443): Max retries exceeded with url: /api/v1/pulses/subscribed?limit=20 (Caused by ResponseError('too many 504 error responses'))

During handling of the above exception, another exception occurred:

RetryError Traceback (most recent call last)
C:\ProgramData\Anaconda3\lib\site-packages\OTXv2.py in get(self, url, **kwargs)
175 headers=self.headers,
--> 176 proxies=self.proxies,
177 )

C:\ProgramData\Anaconda3\lib\site-packages\requests\sessions.py in get(self, url, **kwargs)
545 kwargs.setdefault('allow_redirects', True)
--> 546 return self.request('GET', url, **kwargs)
547

C:\ProgramData\Anaconda3\lib\site-packages\requests\sessions.py in request(self, method, url, params, data, headers, cookies, files, auth, timeout, allow_redirects, proxies, hooks, stream, verify, cert, json)
532 send_kwargs.update(settings)
--> 533 resp = self.send(prep, **send_kwargs)
534

C:\ProgramData\Anaconda3\lib\site-packages\requests\sessions.py in send(self, request, **kwargs)
645 # Send the request
--> 646 r = adapter.send(request, **kwargs)
647

C:\ProgramData\Anaconda3\lib\site-packages\requests\adapters.py in send(self, request, stream, timeout, verify, cert, proxies)
506 if isinstance(e.reason, ResponseError):
--> 507 raise RetryError(e, request=request)
508

RetryError: HTTPSConnectionPool(host='otx.alienvault.com', port=443): Max retries exceeded with url: /api/v1/pulses/subscribed?limit=20 (Caused by ResponseError('too many 504 error responses'))

During handling of the above exception, another exception occurred:

RetryError Traceback (most recent call last)
in
----> 1 pulses = otx.getall()

C:\ProgramData\Anaconda3\lib\site-packages\OTXv2.py in getall(self, modified_since, author_name, limit, max_page, max_items, iter)
393 return self.walkapi(
394 self.create_url(SUBSCRIBED, **args), iter=iter,
--> 395 max_page=max_page, max_items=max_items
396 )
397

C:\ProgramData\Anaconda3\lib\site-packages\OTXv2.py in walkapi(self, url, iter, max_page, max_items, method, body)
372 return self.walkapi_iter(url, max_page=max_page, max_items=max_items, method=method, body=body)
373 else:
--> 374 return list(self.walkapi_iter(url, max_page=max_page, max_items=max_items, method=method, body=body))
375
376 def getall(self, modified_since=None, author_name=None, limit=20, max_page=None, max_items=None, iter=False):

C:\ProgramData\Anaconda3\lib\site-packages\OTXv2.py in walkapi_iter(self, url, max_page, max_items, method, body)
353
354 if method == 'GET':
--> 355 data = self.get(next_page_url)
356 elif method == 'POST':
357 data = self.post(next_page_url, body=body)

C:\ProgramData\Anaconda3\lib\site-packages\OTXv2.py in get(self, url, **kwargs)
178 return self.handle_response_errors(response).json()
179 except requests.exceptions.RetryError:
--> 180 raise RetryError()
181
182 def patch(self, url, body, **kwargs):

RetryError: 'Exceeded maximum number of retries'

Psr18delhi · 2020-08-30T08:31:50Z

I was trying to extract my subscribed pulses but i am not able to download that. I am getting the above error. is there is any limit to download the subscribed or following pulses. I have 50K+ pulses in my dashboard is this is too large to download.

earlier i was able to download the 42K pulses. but this time i am getting the error. Please provide the solution.

rustybrooks · 2021-03-30T00:22:07Z

I didn't see this until now - are you still having problems with it? Can you post some of your code, or send it to me at rbrooks@alienvault.com and I'll take a look? Please be careful not to post code with your API key in it!

Psr18delhi · 2023-03-06T07:13:01Z

@rustybrooks can you please help me to resolve this?

from OTXv2 import OTXv2, IndicatorTypes
from pandas import json_normalize
from datetime import datetime, timedelta, date
import json
import pandas

otx = OTXv2(API Key)
pulses = otx.getall()
events = otx.getsince(timestamp = "2023-02-23")

Till 10th Feb 2023. I was able to use the same functions and get the results. but then after I got the below error message.

Can you please help me to solve this problem?

MaxRetryError Traceback (most recent call last)
~\Anaconda3\lib\site-packages\requests\adapters.py in send(self, request, stream, timeout, verify, cert, proxies)
448 retries=self.max_retries,
--> 449 timeout=timeout
450 )

~\Anaconda3\lib\site-packages\urllib3\connectionpool.py in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
845 body_pos=body_pos,
--> 846 **response_kw
847 )

~\Anaconda3\lib\site-packages\urllib3\connectionpool.py in urlopen(self, method, url, body, headers, retries, redirect, assert_same_host, timeout, pool_timeout, release_conn, chunked, body_pos, **response_kw)
818 try:
--> 819 retries = retries.increment(method, url, response=response, _pool=self)
820 except MaxRetryError:

~\Anaconda3\lib\site-packages\urllib3\util\retry.py in increment(self, method, url, response, error, _pool, _stacktrace)
435 if new_retry.is_exhausted():
--> 436 raise MaxRetryError(_pool, url, error or ResponseError(cause))
437

MaxRetryError: HTTPSConnectionPool(host='otx.alienvault.com', port=443): Max retries exceeded with url: /api/v1/pulses/subscribed?limit=20&modified_since=2023-02-23 (Caused by ResponseError('too many 504 error responses'))

During handling of the above exception, another exception occurred:

RetryError Traceback (most recent call last)
~\Anaconda3\lib\site-packages\OTXv2.py in get(self, url, **kwargs)
175 headers=self.headers,
--> 176 proxies=self.proxies,
177 )

~\Anaconda3\lib\site-packages\requests\sessions.py in get(self, url, **kwargs)
545 kwargs.setdefault('allow_redirects', True)
--> 546 return self.request('GET', url, **kwargs)
547

~\Anaconda3\lib\site-packages\requests\sessions.py in request(self, method, url, params, data, headers, cookies, files, auth, timeout, allow_redirects, proxies, hooks, stream, verify, cert, json)
532 send_kwargs.update(settings)
--> 533 resp = self.send(prep, **send_kwargs)
534

~\Anaconda3\lib\site-packages\requests\sessions.py in send(self, request, **kwargs)
645 # Send the request
--> 646 r = adapter.send(request, **kwargs)
647

~\Anaconda3\lib\site-packages\requests\adapters.py in send(self, request, stream, timeout, verify, cert, proxies)
506 if isinstance(e.reason, ResponseError):
--> 507 raise RetryError(e, request=request)
508

RetryError: HTTPSConnectionPool(host='otx.alienvault.com', port=443): Max retries exceeded with url: /api/v1/pulses/subscribed?limit=20&modified_since=2023-02-23 (Caused by ResponseError('too many 504 error responses'))

During handling of the above exception, another exception occurred:

RetryError Traceback (most recent call last)
in
----> 1 events = otx.getsince(timestamp = "2023-02-23")

~\Anaconda3\lib\site-packages\OTXv2.py in getsince(self, timestamp, limit, max_page, max_items)
418 """
419
--> 420 return self.getall(limit=limit, modified_since=timestamp, max_page=max_page, max_items=max_items, iter=False)
421
422 def getsince_iter(self, timestamp, limit=20, max_page=None, max_items=None):

~\Anaconda3\lib\site-packages\OTXv2.py in getall(self, modified_since, author_name, limit, max_page, max_items, iter)
393 return self.walkapi(
394 self.create_url(SUBSCRIBED, **args), iter=iter,
--> 395 max_page=max_page, max_items=max_items
396 )
397

~\Anaconda3\lib\site-packages\OTXv2.py in walkapi(self, url, iter, max_page, max_items, method, body)
372 return self.walkapi_iter(url, max_page=max_page, max_items=max_items, method=method, body=body)
373 else:
--> 374 return list(self.walkapi_iter(url, max_page=max_page, max_items=max_items, method=method, body=body))
375
376 def getall(self, modified_since=None, author_name=None, limit=20, max_page=None, max_items=None, iter=False):

~\Anaconda3\lib\site-packages\OTXv2.py in walkapi_iter(self, url, max_page, max_items, method, body)
353
354 if method == 'GET':
--> 355 data = self.get(next_page_url)
356 elif method == 'POST':
357 data = self.post(next_page_url, body=body)

~\Anaconda3\lib\site-packages\OTXv2.py in get(self, url, **kwargs)
178 return self.handle_response_errors(response).json()
179 except requests.exceptions.RetryError:
--> 180 raise RetryError()
181
182 def patch(self, url, body, **kwargs):

RetryError: 'Exceeded maximum number of retries'

basvs · 2023-03-06T11:53:51Z

We have the same problem;
library version: OTXv2-1.5.12

Everything worked until February 25th.

requests.exceptions.RetryError: HTTPSConnectionPool(host='otx.alienvault.com', port=443): Max retries exceeded with url: /api/v1/pulses/subscribed?limit=50&modified_since=2023-02-25T23%3A00%3A07.206000 (Caused by ResponseError('too many 504 error responses'))

We tried the library with both limit=20 and limit=50.

We also tried to request the URL with wget without the limit, but with the modified_since. That worked, but the library always sends the limit parameter..

rholloway · 2023-03-13T17:04:55Z

You are not alone @basvs . Not sure what's changed recently but the past couple weeks we have had very unstable results as well. Haven't made any changes on our end but getting lots of 504 recently.

Psr18delhi · 2023-04-28T03:56:34Z

@rustybrooks did you have any update?

rustybrooks · 2023-04-28T04:00:00Z

I haven't worked on OTX in well over a year. I don't actually know who's responsible for the OTX backend infrastructure these days. Sorry.

Psr18delhi · 2023-05-02T09:54:00Z

@chrisdoman any update?

AnduinBrian · 2023-05-19T08:19:10Z

i still have the same problem
my code is simple:

from OTXv2 import OTXv2
from OTXv2 import IndicatorTypes

otx = OTXv2("xxxx")
pulse = otx.getall(max_items=3, limit=5)
print(pulse)

Even i set the limit but still have error: OTXv2.RetryError: 'Exceeded maximum number of retries'
The status code is 504. I think this api might dead lol

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

otx.getall() having error "Exceeded maximum number of retries" #55

otx.getall() having error "Exceeded maximum number of retries" #55

Psr18delhi commented Aug 30, 2020

Psr18delhi commented Aug 30, 2020

rustybrooks commented Mar 30, 2021

Psr18delhi commented Mar 6, 2023

basvs commented Mar 6, 2023

rholloway commented Mar 13, 2023

Psr18delhi commented Apr 28, 2023

rustybrooks commented Apr 28, 2023

Psr18delhi commented May 2, 2023

AnduinBrian commented May 19, 2023 •

edited

Loading

otx.getall() having error "Exceeded maximum number of retries" #55

otx.getall() having error "Exceeded maximum number of retries" #55

Comments

Psr18delhi commented Aug 30, 2020

Psr18delhi commented Aug 30, 2020

rustybrooks commented Mar 30, 2021

Psr18delhi commented Mar 6, 2023

basvs commented Mar 6, 2023

rholloway commented Mar 13, 2023

Psr18delhi commented Apr 28, 2023

rustybrooks commented Apr 28, 2023

Psr18delhi commented May 2, 2023

AnduinBrian commented May 19, 2023 • edited Loading

AnduinBrian commented May 19, 2023 •

edited

Loading