Network load performance between Starlette and Flask with urllib3, requests, and httpx clients

Flask and Starlette performance with urllib3, requests, and httpx

Source code available at https://github.com/aug2uag/starlette-flask-benchmark

The purpose of this benchmark was to evaluate the performance difference under load for a Flask and Starlette microframework with various popular HTTP clients.

These benchmark results were from 1000 cycles.

Results

Flask @ 1000 cycles (testing smaller JSON response)

urllib3:     6.009076181
requests:    8.310135888
httpx:       14.922275728

Flask @ 1000 cycles (testing larger text response)

urllib3:     6.489965677
requests:    7.959935096000001
httpx:       15.216326602

Starlette @ 1000 cycles (testing smaller JSON response)

urllib3:     6.489965677
requests:    7.959935096000001
httpx:       15.216326602

Starlette @ 1000 cycles (testing larger text response)

urllib3:     0.861392250999998
requests:    7.092012402999998
httpx:       12.933335759999999

HTTP libraries

Across the board, the httpx library underperformed relative to requests and urllib3. This might be because the httpx.get method is not benefiting from its async capabilities. The most efficient library of the three was consistently urllib3, followed by the requests library.

The size difference in the request body in this experiment had little or no effect, and indicates the difference in the request bodies was nominal.

Frameworks

Both Flask and Starlette are similar in implementation. Flask is more focused on delivering a full-stack experience, while Starlette is organized to be more biased towards headless services.

The performance of Starlette was modestly more good relative to the requests library. However, Starlette was significantly more performant with urllib3.

Discussion

The urllib3 results for Starlette may be indicating failure, although the server logs looked Ok-- the performance of urllib3 with Starlette is simply unbelievable. Starlette and Python 3 are optimized with subroutines, which may explain the dramatic differences between Flask and Starlette in their performance with urllib3 in case it is working. In that case there are clear multithreading capabilities at work that are stratespherically advantageous to utilize in high performance network I/O applications written in Python. Whereas Flask utilizes WGSI, Starlette implements ASGI that seems to be on its way for taking over Python internet gateways.

Therefore, Starlette with ASGI has superior advantages over Flask for microframework architectures.