Support for multipart/form-data requests (#287)
This commit is contained in:
@@ -43,7 +43,8 @@ describes the backwards incompatible changes that were made.
|
|||||||
The following features are planned for future releases of Microdot, both for
|
The following features are planned for future releases of Microdot, both for
|
||||||
MicroPython and CPython:
|
MicroPython and CPython:
|
||||||
|
|
||||||
- Support for forms encoded in `multipart/form-data` format
|
- Authentication support, similar to [Flask-Login](https://github.com/maxcountryman/flask-login) for Flask (**Added in version 2.1**)
|
||||||
|
- Support for forms encoded in `multipart/form-data` format (**Added in version 2.2**)
|
||||||
- OpenAPI integration, similar to [APIFairy](https://github.com/miguelgrinberg/apifairy) for Flask
|
- OpenAPI integration, similar to [APIFairy](https://github.com/miguelgrinberg/apifairy) for Flask
|
||||||
|
|
||||||
In addition to the above, the following extensions are also under consideration,
|
In addition to the above, the following extensions are also under consideration,
|
||||||
@@ -52,4 +53,4 @@ but only for CPython:
|
|||||||
- Database integration through [SQLAlchemy](https://github.com/sqlalchemy/sqlalchemy)
|
- Database integration through [SQLAlchemy](https://github.com/sqlalchemy/sqlalchemy)
|
||||||
- Socket.IO support through [python-socketio](https://github.com/miguelgrinberg/python-socketio)
|
- Socket.IO support through [python-socketio](https://github.com/miguelgrinberg/python-socketio)
|
||||||
|
|
||||||
Do you have other ideas to propose? Let's [discuss them](https://github.com/miguelgrinberg/microdot/discussions/new?category=ideas)!
|
Do you have other ideas to propose? Let's [discuss them](https://github.com/:miguelgrinberg/microdot/discussions/new?category=ideas)!
|
||||||
|
|||||||
@@ -14,6 +14,12 @@ Core API
|
|||||||
:members:
|
:members:
|
||||||
|
|
||||||
|
|
||||||
|
Multipart Forms
|
||||||
|
---------------
|
||||||
|
|
||||||
|
.. automodule:: microdot.multipart
|
||||||
|
:members:
|
||||||
|
|
||||||
WebSocket
|
WebSocket
|
||||||
---------
|
---------
|
||||||
|
|
||||||
|
|||||||
@@ -5,8 +5,82 @@ Microdot is a highly extensible web application framework. The extensions
|
|||||||
described in this section are maintained as part of the Microdot project in
|
described in this section are maintained as part of the Microdot project in
|
||||||
the same source code repository.
|
the same source code repository.
|
||||||
|
|
||||||
|
Multipart Forms
|
||||||
|
~~~~~~~~~~~~~~~
|
||||||
|
|
||||||
|
.. list-table::
|
||||||
|
:align: left
|
||||||
|
|
||||||
|
* - Compatibility
|
||||||
|
- | CPython & MicroPython
|
||||||
|
|
||||||
|
* - Required Microdot source files
|
||||||
|
- | `multipart.py <https://github.com/miguelgrinberg/microdot/tree/main/src/microdot/multipart.py>`_
|
||||||
|
| `helpers.py <https://github.com/miguelgrinberg/microdot/tree/main/src/microdot/helpers.py>`_
|
||||||
|
|
||||||
|
* - Required external dependencies
|
||||||
|
- | None
|
||||||
|
|
||||||
|
* - Examples
|
||||||
|
- | `formdata.py <https://github.com/miguelgrinberg/microdot/blob/main/examples/uploads/formdata.py>`_
|
||||||
|
|
||||||
|
The multipart extension handles multipart forms, including those that have file
|
||||||
|
uploads.
|
||||||
|
|
||||||
|
The :func:`with_form_data <microdot.multipart.with_form_data>` decorator
|
||||||
|
provides the simplest way to work with these forms. With this decorator added
|
||||||
|
to the route, whenever the client sends a multipart request the
|
||||||
|
:attr:`request.form <microdot.Request.form>` and
|
||||||
|
:attr:`request.files <microdot.Request.files>` properties are populated with
|
||||||
|
the submitted data. For form fields the field values are always strings. For
|
||||||
|
files, they are instances of the
|
||||||
|
:class:`FileUpload <microdot.multipart.FileUpload>` class.
|
||||||
|
|
||||||
|
Example::
|
||||||
|
|
||||||
|
from microdot.multipart import with_form_data
|
||||||
|
|
||||||
|
@app.post('/upload')
|
||||||
|
@with_form_data
|
||||||
|
async def upload(request):
|
||||||
|
print('form fields:', request.form)
|
||||||
|
print('files:', request.files)
|
||||||
|
|
||||||
|
One disadvantage of the ``@with_form_data`` decorator is that it has to copy
|
||||||
|
any uploaded files to memory or temporary disk files, depending on their size.
|
||||||
|
The :attr:`FileUpload.max_memory_size <microdot.multipart.FileUpload.max_memory_size>`
|
||||||
|
attribute can be used to control the cutoff size above which a file upload
|
||||||
|
is transferred to a temporary file.
|
||||||
|
|
||||||
|
A more performant alternative to the ``@with_form_data`` decorator is the
|
||||||
|
:class:`FormDataIter <microdot.multipart.FormDataIter>` class, which iterates
|
||||||
|
over the form fields sequentially, giving the application the option to parse
|
||||||
|
the form fields on the fly and decide what to copy and what to discard. When
|
||||||
|
using ``FormDataIter`` the ``request.form`` and ``request.files`` attributes
|
||||||
|
are not used.
|
||||||
|
|
||||||
|
Example::
|
||||||
|
|
||||||
|
|
||||||
|
from microdot.multipart import FormDataIter
|
||||||
|
|
||||||
|
@app.post('/upload')
|
||||||
|
async def upload(request):
|
||||||
|
async for name, value in FormDataIter(request):
|
||||||
|
print(name, value)
|
||||||
|
|
||||||
|
For fields that contain an uploaded file, the ``value`` returned by the
|
||||||
|
iterator is the same ``FileUpload`` instance. The application can choose to
|
||||||
|
save the file with the :meth:`save() <microdot.multipart.FileUpload.save>`
|
||||||
|
method, or read it with the :meth:`read() <microdot.multipart.FileUpload.read>`
|
||||||
|
method, optionally passing a size to read it in chunks. The
|
||||||
|
:meth:`copy() <microdot.multipart.FileUpload.copy>` method is also available to
|
||||||
|
apply the copying logic used by the ``@with_form_data`` decorator, which is
|
||||||
|
inefficient but allows the file to be set aside to be processed later, after
|
||||||
|
the remaining form fields.
|
||||||
|
|
||||||
WebSocket
|
WebSocket
|
||||||
~~~~~~~~-
|
~~~~~~~~~
|
||||||
|
|
||||||
.. list-table::
|
.. list-table::
|
||||||
:align: left
|
:align: left
|
||||||
@@ -16,6 +90,7 @@ WebSocket
|
|||||||
|
|
||||||
* - Required Microdot source files
|
* - Required Microdot source files
|
||||||
- | `websocket.py <https://github.com/miguelgrinberg/microdot/tree/main/src/microdot/websocket.py>`_
|
- | `websocket.py <https://github.com/miguelgrinberg/microdot/tree/main/src/microdot/websocket.py>`_
|
||||||
|
| `helpers.py <https://github.com/miguelgrinberg/microdot/tree/main/src/microdot/helpers.py>`_
|
||||||
|
|
||||||
* - Required external dependencies
|
* - Required external dependencies
|
||||||
- | None
|
- | None
|
||||||
@@ -32,12 +107,14 @@ messages respectively.
|
|||||||
|
|
||||||
Example::
|
Example::
|
||||||
|
|
||||||
@app.route('/echo')
|
from microdot.websocket import with_websocket
|
||||||
@with_websocket
|
|
||||||
async def echo(request, ws):
|
@app.route('/echo')
|
||||||
while True:
|
@with_websocket
|
||||||
message = await ws.receive()
|
async def echo(request, ws):
|
||||||
await ws.send(message)
|
while True:
|
||||||
|
message = await ws.receive()
|
||||||
|
await ws.send(message)
|
||||||
|
|
||||||
Server-Sent Events
|
Server-Sent Events
|
||||||
~~~~~~~~~~~~~~~~~~
|
~~~~~~~~~~~~~~~~~~
|
||||||
@@ -50,6 +127,7 @@ Server-Sent Events
|
|||||||
|
|
||||||
* - Required Microdot source files
|
* - Required Microdot source files
|
||||||
- | `sse.py <https://github.com/miguelgrinberg/microdot/tree/main/src/microdot/sse.py>`_
|
- | `sse.py <https://github.com/miguelgrinberg/microdot/tree/main/src/microdot/sse.py>`_
|
||||||
|
| `helpers.py <https://github.com/miguelgrinberg/microdot/tree/main/src/microdot/helpers.py>`_
|
||||||
|
|
||||||
* - Required external dependencies
|
* - Required external dependencies
|
||||||
- | None
|
- | None
|
||||||
@@ -65,6 +143,8 @@ asynchronous method to send an event to the client.
|
|||||||
|
|
||||||
Example::
|
Example::
|
||||||
|
|
||||||
|
from microdot.sse import with_sse
|
||||||
|
|
||||||
@app.route('/events')
|
@app.route('/events')
|
||||||
@with_sse
|
@with_sse
|
||||||
async def events(request, sse):
|
async def events(request, sse):
|
||||||
@@ -213,6 +293,7 @@ Secure User Sessions
|
|||||||
|
|
||||||
* - Required Microdot source files
|
* - Required Microdot source files
|
||||||
- | `session.py <https://github.com/miguelgrinberg/microdot/tree/main/src/microdot/session.py>`_
|
- | `session.py <https://github.com/miguelgrinberg/microdot/tree/main/src/microdot/session.py>`_
|
||||||
|
| `helpers.py <https://github.com/miguelgrinberg/microdot/tree/main/src/microdot/helpers.py>`_
|
||||||
|
|
||||||
* - Required external dependencies
|
* - Required external dependencies
|
||||||
- | CPython: `PyJWT <https://pyjwt.readthedocs.io/>`_
|
- | CPython: `PyJWT <https://pyjwt.readthedocs.io/>`_
|
||||||
@@ -369,6 +450,7 @@ User Logins
|
|||||||
* - Required Microdot source files
|
* - Required Microdot source files
|
||||||
- | `login.py <https://github.com/miguelgrinberg/microdot/tree/main/src/microdot/auth.py>`_
|
- | `login.py <https://github.com/miguelgrinberg/microdot/tree/main/src/microdot/auth.py>`_
|
||||||
| `session.py <https://github.com/miguelgrinberg/microdot/tree/main/src/microdot/session.py>`_
|
| `session.py <https://github.com/miguelgrinberg/microdot/tree/main/src/microdot/session.py>`_
|
||||||
|
| `helpers.py <https://github.com/miguelgrinberg/microdot/tree/main/src/microdot/helpers.py>`_
|
||||||
* - Required external dependencies
|
* - Required external dependencies
|
||||||
- | CPython: `PyJWT <https://pyjwt.readthedocs.io/>`_
|
- | CPython: `PyJWT <https://pyjwt.readthedocs.io/>`_
|
||||||
| MicroPython: `jwt.py <https://github.com/micropython/micropython-lib/blob/master/python-ecosys/pyjwt/jwt.py>`_,
|
| MicroPython: `jwt.py <https://github.com/micropython/micropython-lib/blob/master/python-ecosys/pyjwt/jwt.py>`_,
|
||||||
|
|||||||
@@ -1 +1,4 @@
|
|||||||
This directory contains file upload examples.
|
This directory contains file upload examples.
|
||||||
|
|
||||||
|
- `simple_uploads.py` demonstrates how to upload a single file.
|
||||||
|
- `formdata.py` demonstrates how to process a form that includes file uploads.
|
||||||
|
|||||||
17
examples/uploads/formdata.html
Normal file
17
examples/uploads/formdata.html
Normal file
@@ -0,0 +1,17 @@
|
|||||||
|
<!doctype html>
|
||||||
|
<html>
|
||||||
|
<head>
|
||||||
|
<title>Microdot Multipart Form-Data Example</title>
|
||||||
|
<meta charset="UTF-8">
|
||||||
|
</head>
|
||||||
|
<body>
|
||||||
|
<h1>Microdot Multipart Form-Data Example</h1>
|
||||||
|
<form method="POST" action="" enctype="multipart/form-data">
|
||||||
|
<p>Name: <input type="text" name="name" /></p>
|
||||||
|
<p>Age: <input type="text" name="age" /></p>
|
||||||
|
<p>Comments: <textarea name="comments" rows="4"></textarea></p>
|
||||||
|
<p>File: <input type="file" id="file" name="file" /></p>
|
||||||
|
<input type="submit" value="Submit" />
|
||||||
|
</form>
|
||||||
|
</body>
|
||||||
|
</html>
|
||||||
26
examples/uploads/formdata.py
Normal file
26
examples/uploads/formdata.py
Normal file
@@ -0,0 +1,26 @@
|
|||||||
|
from microdot import Microdot, send_file, Request
|
||||||
|
from microdot.multipart import with_form_data
|
||||||
|
|
||||||
|
app = Microdot()
|
||||||
|
Request.max_content_length = 1024 * 1024 # 1MB (change as needed)
|
||||||
|
|
||||||
|
|
||||||
|
@app.get('/')
|
||||||
|
async def index(request):
|
||||||
|
return send_file('formdata.html')
|
||||||
|
|
||||||
|
|
||||||
|
@app.post('/')
|
||||||
|
@with_form_data
|
||||||
|
async def upload(request):
|
||||||
|
print('Form fields:')
|
||||||
|
for field, value in request.form.items():
|
||||||
|
print(f'- {field}: {value}')
|
||||||
|
print('\nFile uploads:')
|
||||||
|
for field, value in request.files.items():
|
||||||
|
print(f'- {field}: {value.filename}, {await value.read()}')
|
||||||
|
return 'We have received your data!'
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == '__main__':
|
||||||
|
app.run(debug=True)
|
||||||
@@ -6,7 +6,7 @@ Request.max_content_length = 1024 * 1024 # 1MB (change as needed)
|
|||||||
|
|
||||||
@app.get('/')
|
@app.get('/')
|
||||||
async def index(request):
|
async def index(request):
|
||||||
return send_file('index.html')
|
return send_file('simple_uploads.html')
|
||||||
|
|
||||||
|
|
||||||
@app.post('/upload')
|
@app.post('/upload')
|
||||||
@@ -1,2 +1,2 @@
|
|||||||
from microdot.microdot import Microdot, Request, Response, abort, redirect, \
|
from microdot.microdot import Microdot, Request, Response, abort, redirect, \
|
||||||
send_file, URLPattern # noqa: F401
|
send_file, URLPattern, AsyncBytesIO, iscoroutine # noqa: F401
|
||||||
|
|||||||
@@ -371,6 +371,7 @@ class Request:
|
|||||||
self.sock = sock
|
self.sock = sock
|
||||||
self._json = None
|
self._json = None
|
||||||
self._form = None
|
self._form = None
|
||||||
|
self._files = None
|
||||||
self.after_request_handlers = []
|
self.after_request_handlers = []
|
||||||
|
|
||||||
@staticmethod
|
@staticmethod
|
||||||
@@ -465,7 +466,13 @@ class Request:
|
|||||||
def form(self):
|
def form(self):
|
||||||
"""The parsed form submission body, as a
|
"""The parsed form submission body, as a
|
||||||
:class:`MultiDict <microdot.MultiDict>` object, or ``None`` if the
|
:class:`MultiDict <microdot.MultiDict>` object, or ``None`` if the
|
||||||
request does not have a form submission."""
|
request does not have a form submission.
|
||||||
|
|
||||||
|
Forms that are URL encoded are processed by default. For multipart
|
||||||
|
forms to be processed, the
|
||||||
|
:func:`with_form_data <microdot.multipart.with_form_data>`
|
||||||
|
decorator must be added to the route.
|
||||||
|
"""
|
||||||
if self._form is None:
|
if self._form is None:
|
||||||
if self.content_type is None:
|
if self.content_type is None:
|
||||||
return None
|
return None
|
||||||
@@ -475,6 +482,17 @@ class Request:
|
|||||||
self._form = self._parse_urlencoded(self.body)
|
self._form = self._parse_urlencoded(self.body)
|
||||||
return self._form
|
return self._form
|
||||||
|
|
||||||
|
@property
|
||||||
|
def files(self):
|
||||||
|
"""The files uploaded in the request as a dictionary, or ``None`` if
|
||||||
|
the request does not have any files.
|
||||||
|
|
||||||
|
The :func:`with_form_data <microdot.multipart.with_form_data>`
|
||||||
|
decorator must be added to the route that receives file uploads for
|
||||||
|
this property to be set.
|
||||||
|
"""
|
||||||
|
return self._files
|
||||||
|
|
||||||
def after_request(self, f):
|
def after_request(self, f):
|
||||||
"""Register a request-specific function to run after the request is
|
"""Register a request-specific function to run after the request is
|
||||||
handled. Request-specific after request handlers run at the very end,
|
handled. Request-specific after request handlers run at the very end,
|
||||||
|
|||||||
291
src/microdot/multipart.py
Normal file
291
src/microdot/multipart.py
Normal file
@@ -0,0 +1,291 @@
|
|||||||
|
import os
|
||||||
|
from random import choice
|
||||||
|
from microdot import abort, iscoroutine, AsyncBytesIO
|
||||||
|
from microdot.helpers import wraps
|
||||||
|
|
||||||
|
|
||||||
|
class FormDataIter:
|
||||||
|
"""Asynchronous iterator that parses a ``multipart/form-data`` body and
|
||||||
|
returns form fields and files as they are parsed.
|
||||||
|
|
||||||
|
:param request: the request object to parse.
|
||||||
|
|
||||||
|
Example usage::
|
||||||
|
|
||||||
|
from microdot.multipart import FormDataIter
|
||||||
|
|
||||||
|
@app.post('/upload')
|
||||||
|
async def upload(request):
|
||||||
|
async for name, value in FormDataIter(request):
|
||||||
|
print(name, value)
|
||||||
|
|
||||||
|
The iterator returns no values when the request has a content type other
|
||||||
|
than ``multipart/form-data``. For a file field, the returned value is of
|
||||||
|
type :class:`FileUpload`, which supports the
|
||||||
|
:meth:`read() <FileUpload.read>` and :meth:`save() <FileUpload.save>`
|
||||||
|
methods. Values for regular fields are provided as strings.
|
||||||
|
|
||||||
|
The request body is read efficiently in chunks of size
|
||||||
|
:attr:`buffer_size <FormDataIter.buffer_size>`. On iterations in which a
|
||||||
|
file field is encountered, the file must be consumed before moving on to
|
||||||
|
the next iteration, as the internal stream stored in ``FileUpload``
|
||||||
|
instances is invalidated at the end of the iteration.
|
||||||
|
"""
|
||||||
|
#: The size of the buffer used to read chunks of the request body.
|
||||||
|
buffer_size = 256
|
||||||
|
|
||||||
|
def __init__(self, request):
|
||||||
|
self.request = request
|
||||||
|
self.buffer = None
|
||||||
|
try:
|
||||||
|
mimetype, boundary = request.content_type.rsplit('; boundary=', 1)
|
||||||
|
except ValueError:
|
||||||
|
return # not a multipart request
|
||||||
|
if mimetype.split(';', 1)[0] == \
|
||||||
|
'multipart/form-data': # pragma: no branch
|
||||||
|
self.boundary = b'--' + boundary.encode()
|
||||||
|
self.extra_size = len(boundary) + 4
|
||||||
|
self.buffer = b''
|
||||||
|
|
||||||
|
def __aiter__(self):
|
||||||
|
return self
|
||||||
|
|
||||||
|
async def __anext__(self):
|
||||||
|
if self.buffer is None:
|
||||||
|
raise StopAsyncIteration
|
||||||
|
|
||||||
|
# make sure we have consumed the previous entry
|
||||||
|
while await self._read_buffer(self.buffer_size) != b'':
|
||||||
|
pass
|
||||||
|
|
||||||
|
# make sure we are at a boundary
|
||||||
|
s = self.buffer.split(self.boundary, 1)
|
||||||
|
if len(s) != 2 or s[0] != b'':
|
||||||
|
abort(400) # pragma: no cover
|
||||||
|
self.buffer = s[1]
|
||||||
|
if self.buffer[:2] == b'--':
|
||||||
|
# we have reached the end
|
||||||
|
raise StopAsyncIteration
|
||||||
|
elif self.buffer[:2] != b'\r\n':
|
||||||
|
abort(400) # pragma: no cover
|
||||||
|
self.buffer = self.buffer[2:]
|
||||||
|
|
||||||
|
# parse the headers of this part
|
||||||
|
name = ''
|
||||||
|
filename = None
|
||||||
|
content_type = None
|
||||||
|
while True:
|
||||||
|
await self._fill_buffer()
|
||||||
|
lines = self.buffer.split(b'\r\n', 1)
|
||||||
|
if len(lines) != 2:
|
||||||
|
abort(400) # pragma: no cover
|
||||||
|
line, self.buffer = lines
|
||||||
|
if line == b'':
|
||||||
|
# we reached the end of the headers
|
||||||
|
break
|
||||||
|
header, value = line.decode().split(':', 1)
|
||||||
|
header = header.lower()
|
||||||
|
value = value.strip()
|
||||||
|
if header == 'content-disposition':
|
||||||
|
parts = value.split(';')
|
||||||
|
if len(parts) < 2 or parts[0] != 'form-data':
|
||||||
|
abort(400) # pragma: no cover
|
||||||
|
for part in parts[1:]:
|
||||||
|
part = part.strip()
|
||||||
|
if part.startswith('name="'):
|
||||||
|
name = part[6:-1]
|
||||||
|
elif part.startswith('filename="'): # pragma: no branch
|
||||||
|
filename = part[10:-1]
|
||||||
|
elif header == 'content-type': # pragma: no branch
|
||||||
|
content_type = value
|
||||||
|
|
||||||
|
if filename is None:
|
||||||
|
# this is a regular form field, so we read the value
|
||||||
|
value = b''
|
||||||
|
while True:
|
||||||
|
v = await self._read_buffer(self.buffer_size)
|
||||||
|
value += v
|
||||||
|
if len(v) < self.buffer_size: # pragma: no branch
|
||||||
|
break
|
||||||
|
return name, value.decode()
|
||||||
|
return name, FileUpload(filename, content_type, self._read_buffer)
|
||||||
|
|
||||||
|
async def _fill_buffer(self):
|
||||||
|
self.buffer += await self.request.stream.read(
|
||||||
|
self.buffer_size + self.extra_size - len(self.buffer))
|
||||||
|
|
||||||
|
async def _read_buffer(self, n=-1):
|
||||||
|
data = b''
|
||||||
|
while n == -1 or len(data) < n:
|
||||||
|
await self._fill_buffer()
|
||||||
|
s = self.buffer.split(self.boundary, 1)
|
||||||
|
data += s[0][:n] if n != -1 else s[0]
|
||||||
|
self.buffer = s[0][n:] if n != -1 else b''
|
||||||
|
if len(s) == 2: # pragma: no branch
|
||||||
|
# the end of this part is in the buffer
|
||||||
|
if len(self.buffer) < 2:
|
||||||
|
# we have read all the way to the end of this part
|
||||||
|
data = data[:-(2 - len(self.buffer))] # remove last "\r\n"
|
||||||
|
self.buffer += self.boundary + s[1]
|
||||||
|
return data
|
||||||
|
return data
|
||||||
|
|
||||||
|
|
||||||
|
class FileUpload:
|
||||||
|
"""Class that represents an uploaded file.
|
||||||
|
|
||||||
|
:param filename: the name of the uploaded file.
|
||||||
|
:param content_type: the content type of the uploaded file.
|
||||||
|
:param read: a coroutine that reads from the uploaded file's stream.
|
||||||
|
|
||||||
|
An uploaded file can be read from the stream using the :meth:`read()`
|
||||||
|
method or saved to a file using the :meth:`save()` method.
|
||||||
|
|
||||||
|
Instances of this class do not normally need to be created directly.
|
||||||
|
"""
|
||||||
|
#: The size at which the file is copied to a temporary file.
|
||||||
|
max_memory_size = 1024
|
||||||
|
|
||||||
|
def __init__(self, filename, content_type, read):
|
||||||
|
self.filename = filename
|
||||||
|
self.content_type = content_type
|
||||||
|
self._read = read
|
||||||
|
self._close = None
|
||||||
|
|
||||||
|
async def read(self, n=-1):
|
||||||
|
"""Read up to ``n`` bytes from the uploaded file's stream.
|
||||||
|
|
||||||
|
:param n: the maximum number of bytes to read. If ``n`` is -1 or not
|
||||||
|
given, the entire file is read.
|
||||||
|
"""
|
||||||
|
return await self._read(n)
|
||||||
|
|
||||||
|
async def save(self, path_or_file):
|
||||||
|
"""Save the uploaded file to the given path or file object.
|
||||||
|
|
||||||
|
:param path_or_file: the path to save the file to, or a file object
|
||||||
|
to which the file is to be written.
|
||||||
|
|
||||||
|
The file is read and written in chunks of size
|
||||||
|
:attr:`FormDataIter.buffer_size`.
|
||||||
|
"""
|
||||||
|
if isinstance(path_or_file, str):
|
||||||
|
f = open(path_or_file, 'wb')
|
||||||
|
else:
|
||||||
|
f = path_or_file
|
||||||
|
while True:
|
||||||
|
data = await self.read(FormDataIter.buffer_size)
|
||||||
|
if not data:
|
||||||
|
break
|
||||||
|
f.write(data)
|
||||||
|
if f != path_or_file:
|
||||||
|
f.close()
|
||||||
|
|
||||||
|
async def copy(self, max_memory_size=None):
|
||||||
|
"""Copy the uploaded file to a temporary file, to allow the parsing of
|
||||||
|
the multipart form to continue.
|
||||||
|
|
||||||
|
:param max_memory_size: the maximum size of the file to keep in memory.
|
||||||
|
If not given, then the class attribute of the
|
||||||
|
same name is used.
|
||||||
|
"""
|
||||||
|
max_memory_size = max_memory_size or FileUpload.max_memory_size
|
||||||
|
buffer = await self.read(max_memory_size)
|
||||||
|
if len(buffer) < max_memory_size:
|
||||||
|
f = AsyncBytesIO(buffer)
|
||||||
|
self._read = f.read
|
||||||
|
return self
|
||||||
|
|
||||||
|
# create a temporary file
|
||||||
|
while True:
|
||||||
|
tmpname = "".join([
|
||||||
|
choice('abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ')
|
||||||
|
for _ in range(12)
|
||||||
|
])
|
||||||
|
try:
|
||||||
|
f = open(tmpname, 'x+b')
|
||||||
|
except OSError as e: # pragma: no cover
|
||||||
|
if e.errno == 17:
|
||||||
|
# EEXIST
|
||||||
|
continue
|
||||||
|
elif e.errno == 2:
|
||||||
|
# ENOENT
|
||||||
|
# some MicroPython platforms do not support mode "x"
|
||||||
|
f = open(tmpname, 'w+b')
|
||||||
|
if f.read(1) != b'':
|
||||||
|
f.close()
|
||||||
|
continue
|
||||||
|
else:
|
||||||
|
raise
|
||||||
|
break
|
||||||
|
f.write(buffer)
|
||||||
|
await self.save(f)
|
||||||
|
f.seek(0)
|
||||||
|
|
||||||
|
async def read(n=-1):
|
||||||
|
return f.read(n)
|
||||||
|
|
||||||
|
async def close():
|
||||||
|
f.close()
|
||||||
|
os.remove(tmpname)
|
||||||
|
|
||||||
|
self._read = read
|
||||||
|
self._close = close
|
||||||
|
return self
|
||||||
|
|
||||||
|
async def close(self):
|
||||||
|
"""Close an open file.
|
||||||
|
|
||||||
|
This method must be called to free memory or temporary files created by
|
||||||
|
the ``copy()`` method.
|
||||||
|
|
||||||
|
Note that when using the ``@with_form_data`` decorator this method is
|
||||||
|
called automatically when the request ends.
|
||||||
|
"""
|
||||||
|
if self._close:
|
||||||
|
await self._close()
|
||||||
|
self._close = None
|
||||||
|
|
||||||
|
|
||||||
|
def with_form_data(f):
|
||||||
|
"""Decorator that parses a ``multipart/form-data`` body and updates the
|
||||||
|
request object with the parsed form fields and files.
|
||||||
|
|
||||||
|
Example usage::
|
||||||
|
|
||||||
|
from microdot.multipart import with_form_data
|
||||||
|
|
||||||
|
@app.post('/upload')
|
||||||
|
@with_form_data
|
||||||
|
async def upload(request):
|
||||||
|
print('form fields:', request.form)
|
||||||
|
print('files:', request.files)
|
||||||
|
|
||||||
|
Note: this decorator calls the :meth:`FileUpload.copy()
|
||||||
|
<microdot.multipart.FileUpload.copy>` method on all uploaded files, so that
|
||||||
|
the request can be parsed in its entirety. The files are either copied to
|
||||||
|
memory or a temporary file, depending on their size. The temporary files
|
||||||
|
are automatically deleted when the request ends.
|
||||||
|
"""
|
||||||
|
@wraps(f)
|
||||||
|
async def wrapper(request, *args, **kwargs):
|
||||||
|
form = {}
|
||||||
|
files = {}
|
||||||
|
async for name, value in FormDataIter(request):
|
||||||
|
if isinstance(value, FileUpload):
|
||||||
|
files[name] = await value.copy()
|
||||||
|
else:
|
||||||
|
form[name] = value
|
||||||
|
if form or files:
|
||||||
|
request._form = form
|
||||||
|
request._files = files
|
||||||
|
try:
|
||||||
|
ret = f(request, *args, **kwargs)
|
||||||
|
if iscoroutine(ret):
|
||||||
|
ret = await ret
|
||||||
|
finally:
|
||||||
|
if request.files:
|
||||||
|
for file in request.files.values():
|
||||||
|
await file.close()
|
||||||
|
return ret
|
||||||
|
return wrapper
|
||||||
@@ -4,6 +4,7 @@ from tests.test_request import * # noqa: F401, F403
|
|||||||
from tests.test_response import * # noqa: F401, F403
|
from tests.test_response import * # noqa: F401, F403
|
||||||
from tests.test_urlencode import * # noqa: F401, F403
|
from tests.test_urlencode import * # noqa: F401, F403
|
||||||
from tests.test_url_pattern import * # noqa: F401, F403
|
from tests.test_url_pattern import * # noqa: F401, F403
|
||||||
|
from tests.test_multipart import * # noqa: F401, F403
|
||||||
from tests.test_websocket import * # noqa: F401, F403
|
from tests.test_websocket import * # noqa: F401, F403
|
||||||
from tests.test_sse import * # noqa: F401, F403
|
from tests.test_sse import * # noqa: F401, F403
|
||||||
from tests.test_cors import * # noqa: F401, F403
|
from tests.test_cors import * # noqa: F401, F403
|
||||||
|
|||||||
192
tests/test_multipart.py
Normal file
192
tests/test_multipart.py
Normal file
@@ -0,0 +1,192 @@
|
|||||||
|
import asyncio
|
||||||
|
import os
|
||||||
|
import unittest
|
||||||
|
from microdot import Microdot
|
||||||
|
from microdot.multipart import with_form_data, FileUpload, FormDataIter
|
||||||
|
from microdot.test_client import TestClient
|
||||||
|
|
||||||
|
|
||||||
|
class TestMultipart(unittest.TestCase):
|
||||||
|
@classmethod
|
||||||
|
def setUpClass(cls):
|
||||||
|
if hasattr(asyncio, 'set_event_loop'):
|
||||||
|
asyncio.set_event_loop(asyncio.new_event_loop())
|
||||||
|
cls.loop = asyncio.get_event_loop()
|
||||||
|
|
||||||
|
def _run(self, coro):
|
||||||
|
return self.loop.run_until_complete(coro)
|
||||||
|
|
||||||
|
def test_simple_form(self):
|
||||||
|
app = Microdot()
|
||||||
|
|
||||||
|
@app.post('/sync')
|
||||||
|
@with_form_data
|
||||||
|
def sync_route(req):
|
||||||
|
return dict(req.form)
|
||||||
|
|
||||||
|
@app.post('/async')
|
||||||
|
@with_form_data
|
||||||
|
async def async_route(req):
|
||||||
|
return dict(req.form)
|
||||||
|
|
||||||
|
client = TestClient(app)
|
||||||
|
|
||||||
|
res = self._run(client.post(
|
||||||
|
'/sync', headers={
|
||||||
|
'Content-Type': 'multipart/form-data; boundary=boundary',
|
||||||
|
},
|
||||||
|
body=(
|
||||||
|
b'--boundary\r\n'
|
||||||
|
b'Content-Disposition: form-data; name="foo"\r\n\r\nbar\r\n'
|
||||||
|
b'--boundary\r\n'
|
||||||
|
b'Content-Disposition: form-data; name="baz"\r\n\r\nbaz\r\n'
|
||||||
|
b'--boundary--\r\n')
|
||||||
|
))
|
||||||
|
self.assertEqual(res.status_code, 200)
|
||||||
|
self.assertEqual(res.json, {'foo': 'bar', 'baz': 'baz'})
|
||||||
|
|
||||||
|
res = self._run(client.post(
|
||||||
|
'/async', headers={
|
||||||
|
'Content-Type': 'multipart/form-data; boundary=boundary',
|
||||||
|
},
|
||||||
|
body=(
|
||||||
|
b'--boundary\r\n'
|
||||||
|
b'Content-Disposition: form-data; name="foo"\r\n\r\nbar\r\n'
|
||||||
|
b'--boundary\r\n'
|
||||||
|
b'Content-Disposition: form-data; name="baz"\r\n\r\nbaz\r\n'
|
||||||
|
b'--boundary--\r\n')
|
||||||
|
))
|
||||||
|
self.assertEqual(res.status_code, 200)
|
||||||
|
self.assertEqual(res.json, {'foo': 'bar', 'baz': 'baz'})
|
||||||
|
|
||||||
|
def test_form_with_files(self):
|
||||||
|
saved_max_memory_size = FileUpload.max_memory_size
|
||||||
|
FileUpload.max_memory_size = 5
|
||||||
|
|
||||||
|
app = Microdot()
|
||||||
|
|
||||||
|
@app.post('/async')
|
||||||
|
@with_form_data
|
||||||
|
async def async_route(req):
|
||||||
|
d = dict(req.form)
|
||||||
|
for name, file in req.files.items():
|
||||||
|
d[name] = '{}|{}|{}'.format(file.filename, file.content_type,
|
||||||
|
(await file.read()).decode())
|
||||||
|
return d
|
||||||
|
|
||||||
|
client = TestClient(app)
|
||||||
|
|
||||||
|
res = self._run(client.post(
|
||||||
|
'/async', headers={
|
||||||
|
'Content-Type': 'multipart/form-data; boundary=boundary',
|
||||||
|
},
|
||||||
|
body=(
|
||||||
|
b'--boundary\r\n'
|
||||||
|
b'Content-Disposition: form-data; name="foo"\r\n\r\nbar\r\n'
|
||||||
|
b'--boundary\r\n'
|
||||||
|
b'Content-Disposition: form-data; name="f"; filename="f"\r\n'
|
||||||
|
b'Content-Type: text/plain\r\n\r\nbaz\r\n'
|
||||||
|
b'--boundary\r\n'
|
||||||
|
b'Content-Disposition: form-data; name="g"; filename="g"\r\n'
|
||||||
|
b'Content-Type: text/html\r\n\r\n<p>hello</p>\r\n'
|
||||||
|
b'--boundary\r\n'
|
||||||
|
b'Content-Disposition: form-data; name="x"\r\n\r\ny\r\n'
|
||||||
|
b'--boundary--\r\n')
|
||||||
|
))
|
||||||
|
self.assertEqual(res.status_code, 200)
|
||||||
|
self.assertEqual(res.json, {'foo': 'bar', 'x': 'y',
|
||||||
|
'f': 'f|text/plain|baz',
|
||||||
|
'g': 'g|text/html|<p>hello</p>'})
|
||||||
|
FileUpload.max_memory_size = saved_max_memory_size
|
||||||
|
|
||||||
|
def test_file_save(self):
|
||||||
|
app = Microdot()
|
||||||
|
|
||||||
|
@app.post('/async')
|
||||||
|
@with_form_data
|
||||||
|
async def async_route(req):
|
||||||
|
for _, file in req.files.items():
|
||||||
|
await file.save('_x.txt')
|
||||||
|
|
||||||
|
client = TestClient(app)
|
||||||
|
|
||||||
|
res = self._run(client.post(
|
||||||
|
'/async', headers={
|
||||||
|
'Content-Type': 'multipart/form-data; boundary=boundary',
|
||||||
|
},
|
||||||
|
body=(
|
||||||
|
b'--boundary\r\n'
|
||||||
|
b'Content-Disposition: form-data; name="foo"\r\n\r\nbar\r\n'
|
||||||
|
b'--boundary\r\n'
|
||||||
|
b'Content-Disposition: form-data; name="f"; filename="f"\r\n'
|
||||||
|
b'Content-Type: text/plain\r\n\r\nbaz\r\n'
|
||||||
|
b'--boundary--\r\n')
|
||||||
|
))
|
||||||
|
self.assertEqual(res.status_code, 204)
|
||||||
|
with open('_x.txt', 'rb') as f:
|
||||||
|
self.assertEqual(f.read(), b'baz')
|
||||||
|
os.unlink('_x.txt')
|
||||||
|
|
||||||
|
def test_no_form(self):
|
||||||
|
app = Microdot()
|
||||||
|
|
||||||
|
@app.post('/async')
|
||||||
|
@with_form_data
|
||||||
|
async def async_route(req):
|
||||||
|
return str(req.form)
|
||||||
|
|
||||||
|
client = TestClient(app)
|
||||||
|
|
||||||
|
res = self._run(client.post('/async', body={'foo': 'bar'}))
|
||||||
|
self.assertEqual(res.status_code, 200)
|
||||||
|
self.assertEqual(res.text, 'None')
|
||||||
|
|
||||||
|
def test_upload_iterator(self):
|
||||||
|
app = Microdot()
|
||||||
|
|
||||||
|
@app.post('/async')
|
||||||
|
async def async_route(req):
|
||||||
|
d = {}
|
||||||
|
async for name, value in FormDataIter(req):
|
||||||
|
if isinstance(value, FileUpload):
|
||||||
|
d[name] = '{}|{}|{}'.format(value.filename,
|
||||||
|
value.content_type,
|
||||||
|
(await value.read(4)).decode())
|
||||||
|
else:
|
||||||
|
d[name] = value
|
||||||
|
return d
|
||||||
|
|
||||||
|
client = TestClient(app)
|
||||||
|
|
||||||
|
res = self._run(client.post(
|
||||||
|
'/async', headers={
|
||||||
|
'Content-Type': 'multipart/form-data; boundary=boundary',
|
||||||
|
},
|
||||||
|
body=(
|
||||||
|
b'--boundary\r\n'
|
||||||
|
b'Content-Disposition: form-data; name="foo"\r\n\r\nbar\r\n'
|
||||||
|
b'--boundary\r\n'
|
||||||
|
b'Content-Disposition: form-data; name="f"; filename="f"\r\n'
|
||||||
|
b'Content-Type: text/plain\r\n\r\nbaz\r\n'
|
||||||
|
b'--boundary\r\n'
|
||||||
|
b'Content-Disposition: form-data; name="g"; filename="g.h"\r\n'
|
||||||
|
b'Content-Type: text/html\r\n\r\n<p>hello</p>\r\n'
|
||||||
|
b'--boundary\r\n'
|
||||||
|
b'Content-Disposition: form-data; name="x"\r\n\r\ny\r\n'
|
||||||
|
b'--boundary\r\n'
|
||||||
|
b'Content-Disposition: form-data; name="h"; filename="hh"\r\n'
|
||||||
|
b'Content-Type: text/plain\r\n\r\nyy' + (b'z' * 500) + b'\r\n'
|
||||||
|
b'--boundary\r\n'
|
||||||
|
b'Content-Disposition: form-data; name="i"; filename="i.1"\r\n'
|
||||||
|
b'Content-Type: text/plain\r\n\r\n1234\r\n'
|
||||||
|
b'--boundary--\r\n')
|
||||||
|
))
|
||||||
|
self.assertEqual(res.status_code, 200)
|
||||||
|
self.assertEqual(res.json, {
|
||||||
|
'foo': 'bar',
|
||||||
|
'f': 'f|text/plain|baz',
|
||||||
|
'g': 'g.h|text/html|<p>h',
|
||||||
|
'x': 'y',
|
||||||
|
'h': 'hh|text/plain|yyzz',
|
||||||
|
'i': 'i.1|text/plain|1234',
|
||||||
|
})
|
||||||
Reference in New Issue
Block a user