Jekyll2019-11-14T23:44:27+01:00/feed.xmlPiyush AgrawalA lazy developer on a mission to make life easy.
poushWelcome to new design!2019-11-12T00:00:00+01:002019-11-12T00:00:00+01:00/2019/11/12/welcome-to-new-design<p>Bonjour,</p>
<p>If you are at this page, you are on the new theme of my page. This still has some minor bugs as usual đ. Feel free to leave your feedback or report issues. I would love to read them. If you have forked the old design, then you will have to rebase now on from the repo: <a href="http://github.com/poush/ipiyush.com">github.com/poush/ipiyush.com</a>. The old repo, poush.github.io, now only hosts the compiled jekyll website.</p>
<p>~ Piyush Agrawal</p>poushBonjour,Reset password in Open Even API Server2017-09-12T00:00:00+02:002017-09-12T00:00:00+02:00/Reset-password-in-Open-Even-API-Server<p><img src="/public/AWUmd00BBnleDnbcf1Cfw_img_0.png" alt="image alt text" /></p>
<p>Reset password in Open Even API Server</p>
<p>The addition of reset password API in the <a href="https://github.com/fossasia/open-event-orga-server">Open Event API Server</a> enables the user to send a forgot password request to the server so that user can reset the password.</p>
<p>Reset Password API is a two step process. The first endpoint allows you to request a token to reset the password and this token is sent to the user via email. The second process is making a PATCH request with the token and new password to set the new password on userâs account.</p>
<h3 id="creating-a-reset-token">Creating a Reset token</h3>
<p>This endpoint is non JSON spec based API. A reset token is simply a hash of random bits which is stored in a specific column of userâs table.</p>
<p>hash_ = random.getrandbits(128)</p>
<p>self.reset_password = str(hash_)</p>
<p>Once the user completed the resetting of the password using the specific token, the old token is flushed and the new token is generated. These tokens are all one time use only.</p>
<h3 id="requesting-a-token">Requesting a token</h3>
<p>A token can be request on a specific endpoint <strong><em>POST /v1/auth/reset-password</em></strong></p>
<p>The token with the direct link will be sent to registered email.</p>
<p>link = make_frontend_url(â/reset-passwordâ, {âtokenâ: user.reset_password})</p>
<p>send_email_with_action(user, PASSWORD_RESET,</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> app_name=get_settings()['app_name'], link=link)
</code></pre></div></div>
<h3 id="flow-with-frontend">Flow with frontend</h3>
<p>The flow is broken into 2 steps with front end is serving to the backend. The user when click on forget password will be redirected to reset password page in front end which will call the API endpoint in the backend with an email to send the token.</p>
<p>The email received will contain the link for the front end URL which when clicked will redirect the user to the front end page of providing the new password. The new password entered with the token will be sent to API server by front end and reset password will complete.</p>
<h3 id="updating-password">Updating Password</h3>
<p>Once clicked on the link in the email, the user will be asked to provide the new password. This password will be sent to the endpoint <strong><em>PATCH /v1/auth/reset-password</em></strong></p>
<p>The body will receive the token and the new password to update. The user will be identified using the token and password is updated for the respective user.</p>
<p>try:</p>
<p>user = User.query.filter_by(reset_password=token).one()</p>
<p>except NoResultFound:</p>
<p>return abort(</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> make_response(jsonify(error="User not found"), 404)
</code></pre></div></div>
<p>)</p>
<p>else:</p>
<p>user.password = password</p>
<p>save_to_db(user)</p>
<h3 id="references">References</h3>
<ol>
<li>
<p>Understand Self-service reset password
<a href="https://en.wikipedia.org/wiki/Self-service_password_reset">https://en.wikipedia.org/wiki/Self-service_password_reset</a></p>
</li>
<li>
<p>Python - getrandbits()
<a href="https://docs.python.org/2/library/random.html">https://docs.python.org/2/library/random.html</a></p>
</li>
</ol>poushGenerating Ticket PDFs in Open Event API Server2017-09-12T00:00:00+02:002017-09-12T00:00:00+02:00/Generating-Ticket-PDFs-in-Open-Event-API-Server<p><img src="/public/rvHPVxaNuPA2Zs7PARrg_img_0.png" alt="image alt text" /></p>
<p>Generating Ticket PDFs in Open Event API Server</p>
<p>In the ordering system of <a href="https://github.com/fossasia/open-event-orga-server">Open Event API Server</a>, there is requirement to send email notifications to the attendees. These attendees receives the url of the pdf of generated ticket. On creating the order, first the pdfs are generated and stored in the preferred storage location and then these are sent to the users through the email.</p>
<p>Generating PDF is a simple process, using <strong><a href="https://github.com/xhtml2pdf/xhtml2pdf">xhtml2pd**f</a></strong> **we can generate PDFs from the html. The generated pdf is then passed to storage helpers to store it in the desired location and pdf-url is updated in the attendees record.</p>
<h3 id="sample-pdf">Sample PDF</h3>
<p>(just the sample ticket of sample event)</p>
<p><img src="/public/rvHPVxaNuPA2Zs7PARrg_img_1.png" alt="image alt text" /></p>
<h3 id="pdf-template">PDF Template</h3>
<p>The templates are written in html which are then converted using the module <a href="https://github.com/xhtml2pdf/xhtml2pdf">xhtml2pdf</a>.</p>
<p>To store the templates a new directory was created at * <strong>*_app/templates _</strong>where all html files are stored. Now, The template directory needs to be updated at flask initializing app so that template engine can pick the templates from there. So in app/<strong>init</strong>.py we updated flask initialization with</p>
<p>template_dir = os.path.dirname(<strong>file</strong>) + â/templatesâ</p>
<p>app = Flask(<strong>name</strong>, static_folder=static_dir, template_folder=template_dir)</p>
<p>This allows the template engine to pick the templates files from this template directory.</p>
<h3 id="generating-pdfs">Generating PDFs</h3>
<p>Generating PDF is done by rendering the html template first. This html content is then parsed into the pdf</p>
<p>file = open(dest, âwbâ)</p>
<p>pisa.CreatePDF(cStringIO.StringIO(pdf_data.encode(âutf-8â)), file)</p>
<p>file.close()</p>
<p>The generated pdf is stored in the temporary location and then passed to storage helper to upload it.</p>
<p>uploaded_file = UploadedFile(dest, filename)</p>
<p>upload_path = UPLOAD_PATHS[âpdfâ][âticket_attendeeâ].format(identifier=get_file_name())</p>
<p>new_file = upload(uploaded_file, upload_path)</p>
<p>This generated pdf path is returned here</p>
<h3 id="rendering-html-and-storing-pdf">Rendering HTML and storing PDF</h3>
<p>for holder in order.ticket_holders:</p>
<p>if holder.id != current_user.id:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> pdf = create_save_pdf(render_template('/pdf/ticket_attendee.html', order=order, holder=holder))
</code></pre></div></div>
<p>else:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> pdf = create_save_pdf(render_template('/pdf/ticket_purchaser.html', order=order))
</code></pre></div></div>
<p>holder.pdf_url = pdf</p>
<p>save_to_db(holder)</p>
<p>The html is rendered using flask template engine and passed to <strong>create_save_pdf</strong> and link is updated on the attendee record.</p>
<h3 id="sending-pdf-on-email">Sending PDF on email</h3>
<p>These pdfs are sent as a link to the email after creating the order. Thus a ticket is sent to each attendee and a summarized order details with attendees to the purchased.</p>
<p>send_email(</p>
<p>to=holder.email,</p>
<p>action=TICKET_PURCHASED_ATTENDEE,</p>
<p>subject=MAILS[TICKET_PURCHASED_ATTENDEE][âsubjectâ].format(</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> event_name=order.event.name,
invoice_id=order.invoice_number
</code></pre></div></div>
<p>),</p>
<p>html= MAILS[TICKET_PURCHASED_ATTENDEE][âmessageâ].format(</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> pdf_url=holder.pdf_url,
event_name=order.event.name
</code></pre></div></div>
<p>)</p>
<p>)</p>
<h3 id="references">References</h3>
<ol>
<li>
<p>Readme - xhtml2pdf
<a href="https://github.com/xhtml2pdf/xhtml2pdf/blob/master/README.rst">https://github.com/xhtml2pdf/xhtml2pdf/blob/master/README.rst</a></p>
</li>
<li>
<p>Using xhtml2pdf and create pdfs
<a href="https://micropyramid.com/blog/generating-pdf-files-in-python-using-xhtml2pdf/">https://micropyramid.com/blog/generating-pdf-files-in-python-using-xhtml2pdf/</a></p>
</li>
</ol>poushA guide to use Permission Manager in Open Event API Server2017-09-12T00:00:00+02:002017-09-12T00:00:00+02:00/A-guide-to-use-Permission-Manager-in-Open-Event-API-Server<p><img src="/public/ZnPzuLKvhpUpxOTCk05cg_img_0.png" alt="image alt text" />A guide to use Permission Manager in Open Event API Server</p>
<p>This article provides a simple guide to use permission manager in <a href="http://github.com/fossasia/open-event-orga-server">Open Event API Server</a>. Permission manager is constantly being improved and new features are being added into it. To ensure that all co-developers get to know about it and make use of them, this blog posts describes every part of permission manager.</p>
<h3 id="bootstrapping">Bootstrapping</h3>
<p>Permission manager as a part of flask-rest-jsonapi works as a decorator for different resources of the API. There are two ways to provide the permission decorator to any view</p>
<ul>
<li>
<p>First one is to provide it in the list of decorators</p>
<ul>
<li>
<p>decorators = (api.has_permission(âis_coorganizerâ, fetch=âevent_idâ,</p>
</li>
<li>
<p>fetch_as=âevent_idâ, model=StripeAuthorization),)</p>
</li>
</ul>
</li>
<li>
<p>Second way is to explicitly provide it as a decorator to any view</p>
<ul>
<li><strong>@api.has_permission</strong>(âcustom_argâ, custom_kwargs=âcustom_kwargsâ)
<strong>def</strong> get(*args, <strong>kwargs):
**return</strong> âHello world !â</li>
</ul>
</li>
</ul>
<p>In the process of booting up, we first need to understand the flow of Resources in API. All resources even before doing any schema check, call the decorators. So this way you will not get any request data in the permission methods. All you will receive is a <strong>dict</strong> of the URL parameters but again it will not include the filter parameters.</p>
<p>Permission Manager receives five parameters as:</p>
<p><strong>def **</strong>permission_manager**(view, view_args, view_kwargs, *args, **kwargs):</p>
<p>First three are provided into it implicitly by <a href="https://flask-rest-jsonapi.readthedocs.io/">flask-rest-jsonapi</a> module</p>
<ul>
<li>
<p><strong>v</strong><strong>iew: **This is the resourceâs view method which is called through the API. For example if I go to */events *then the **get</strong> method of <strong>ResourceList</strong> will be called.</p>
</li>
<li>
<p><strong>v</strong>**iew_args: **These are args associated with that view</p>
</li>
<li>
<p><strong>v</strong>**iew_kwargs: **These are kwargs associated with that resource view. It includes all your URL parameters as well</p>
</li>
<li>
<p><strong>a</strong><strong>rgs: **These are the custom args which are provided when calling the permission manager. Here at permission manager is it expected that the first index of **args</strong> will be the name of permission to check for.</p>
</li>
<li>
<p><strong>k</strong>**wargs: **This is the custom dict which is provided on calling the permission manager. The main pillar of the permission manager. Described below in usage.</p>
</li>
</ul>
<h3 id="using-permission-manager">Using Permission Manager</h3>
<p>Using permission manager is basically understanding the different options you can send through the <strong>kwargs</strong> so here is the list of the things you can send to permission manager</p>
<p>These are all described in the order of priority in permission manager</p>
<ol>
<li>
<p><strong>m</strong><strong>ethod (string)</strong>: You can provide a string containing the methods where permission needs to be checked as comma separated values of different methods in a string.
For example: <strong><em>method=</em></strong><em>âGET,POSTâ</em></p>
</li>
<li>
<p><strong>l</strong>**eave_if (lambda): **This receives a lambda function which should return boolean values. Based on returned value if is true then **it will skip the permission check. **The provided lambda function receives only parameter, *âview_kwargsâ
*Example use case can be the situation where you can leave the permission for any specific related endpoint to some resource and would like to do a manual check in the method itself.</p>
</li>
<li>
<p><strong>c</strong><strong>heck (lambda): **Opposite to leave_if. It receives a lambda function that will return boolean values. Based on returned value, If it is true then only it will go further and check the request for permissions else **will throw forbidden error.</strong></p>
</li>
<li>
<p><strong>f</strong><strong>etch (string): **This is the string containing the name of the key which has to be fetched for **fetch_as **key (described below). Permission manager will first look for this value in **view_kwargs</strong> dict object. If it is not there then it will make the query to get one(described below at <strong>model )</strong></p>
</li>
<li>
<p><strong>f</strong><strong>etch_as (string): **This is the string containing the name of a key. The value of **fetch</strong> key will be sent to the permission functions by this name.</p>
</li>
<li>
<p><strong>m</strong><strong>odel (string): **This is one most interesting concept here. To get the value of **fetch **key. Permission manager first looks into **view_kwargs **and if there no such value then you can still get one through the model. The model attribute here receives the class of the database model which will be used to get the value of **fetch</strong> key.
It makes the query to get the single resource from this model and look for the value of <strong>fetch</strong> key and then pass it to the permission functions/methods.</p>
</li>
</ol>
<p>The interesting part is that by default it uses <*id> *from view_kwargs to get the resource from the model but in any case if there is no specific ID with name <id> on the view_kwargs. You can use these two options as:</id></p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>1. **f****etch_key_url (string): **This is the name of the key whose value will be fetched from view_kwargs and will be used to match through the records in database model to get the resource.
2. **f****etch_key_model (string):** This is the name of the match column in the database model for the **fetch_key_url**, The value of it will be matched with column named as the value of **fetch_key_model.**
</code></pre></div></div>
<p><strong>In case there is no record found in the model then permission manager will throw NotFound 404 Error.</strong></p>
<h3 id="a-helper-for-permissions">A helper for permissions</h3>
<p>The next big thing in permission manager is the addition of new helper function âhas_accessâ</p>
<p><strong>def **</strong>has_access**(access_level, **kwargs):</p>
<p>**if **access_level **in **permissions:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> auth = permissions[access_level](**lambda ***a, **b: True, (), {}, (), **kwargs)
**if **type(auth) **is **bool **and **auth **is **True:
**return **True
</code></pre></div></div>
<p>**return **False</p>
<p>This method allows you to check the permission at the mid of any method of any view and of any resource. Just provide the name of permission in the first parameter and then the additional options needed by the permission function as the **kwargs **values.</p>
<p><strong>This does not throw any exception</strong>. Just returns the boolean value so take care of throwing any exception by yourselves.</p>
<h3 id="anything-to-improve-on">Anything to improve on?</h3>
<p>I will not say this exactly as improvement but I would really like to make it more meaningful and interesting to add permission. May be something like this below:</p>
<p>permission = âMust be co_organizer OR track_organizer, fetch event_id as event_id, use model Eventâ</p>
<p>This clearly needs time to make it. But I see this as an interesting way to add permission. Just provide meaningful text and rest leave it to the permission manager.</p>
<h3 id="resources">Resources:</h3>
<ul>
<li>Permission manager - Flask-rest-jsonapi module
<a href="http://flask-rest-jsonapi.readthedocs.io/en/latest/permission.html">http://flask-rest-jsonapi.readthedocs.io/en/latest/permission.html</a></li>
</ul>poushA guide to use Permission Manager in Open Event API ServerDecorators in Open Event API Server2017-07-04T00:00:00+02:002017-07-04T00:00:00+02:00/2017/07/04/decorators-in-open-event-api-server<p>One of the interesting features of Python is the decorator. Decorators dynamically alter the functionality of a function, method, or class without having to directly use subclasses or change the source code of the function being decorated.
<a href="http://github.com/fossasia/open-event-orga-server">Open Event API Server</a> makes use of decorator in various ways. The ability to wrap a function and run the decorator(s) before executing that function solves various purpose in Python. Earlier before decoupling of Orga Server into API Server and Frontend, decorators were being used for routes, permissions, validations and more.</p>
<p>Now, The API Server mainly uses decorators for:</p>
<ul>
<li>Permissions</li>
<li>Filtering on the basis of view_kwargs or injecting something into view_kwargs</li>
<li>Validations</li>
</ul>
<p>We will take here about first two because validations are simple and we are using them out of the box from <a href="http://marshmallow-jsonapi.readthedocs.io/en/latest/">marshmall-api</a>
The second one is custom implementation made to ensure no separate generic helpers are called which can add additional database queries and call overheads in some scenarios.</p>
<p><strong>Permissions Using Decorators</strong></p>
<p>Flask-rest-jsonapi provides an easy way to add decorators to <code class="highlighter-rouge">Resources</code>. This is as easy as defining this into Resource class</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code>
<span class="n">decorators</span> <span class="o">=</span> <span class="p">(</span><span class="n">some_decorator</span><span class="p">,</span> <span class="p">)</span>
</code></pre></div></div>
<p>On working to event role decorators to use here, I need to follow only these 3 rules</p>
<ul>
<li>If the user is admin or super admin, he/she has full access to all event roles</li>
<li>Then check the userâs role for the given event</li>
<li>Returns the requested resourceâs view if authorized unless returns Forbidden Error response.</li>
</ul>
<p>One of the example is:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">is_organizer</span><span class="p">(</span><span class="n">view</span><span class="p">,</span> <span class="n">view_args</span><span class="p">,</span> <span class="n">view_kwargs</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
<span class="n">user</span> <span class="o">=</span> <span class="n">current_identity</span>
<span class="k">if</span> <span class="n">user</span><span class="o">.</span><span class="n">is_staff</span><span class="p">:</span>
<span class="k">return</span> <span class="n">view</span><span class="p">(</span><span class="o">*</span><span class="n">view_args</span><span class="p">,</span> <span class="o">**</span><span class="n">view_kwargs</span><span class="p">)</span>
<span class="k">if</span> <span class="ow">not</span> <span class="n">user</span><span class="o">.</span><span class="n">is_organizer</span><span class="p">(</span><span class="n">kwargs</span><span class="p">[</span><span class="s">'event_id'</span><span class="p">]):</span>
<span class="k">return</span> <span class="n">ForbiddenError</span><span class="p">({</span><span class="s">'source'</span><span class="p">:</span> <span class="s">''</span><span class="p">},</span> <span class="s">'Organizer access is required'</span><span class="p">)</span><span class="o">.</span><span class="n">respond</span><span class="p">()</span>
<span class="k">return</span> <span class="n">view</span><span class="p">(</span><span class="o">*</span><span class="n">view_args</span><span class="p">,</span> <span class="o">**</span><span class="n">view_kwargs</span><span class="p">)</span>
</code></pre></div></div>
<p>From above example, it is clear that it is following those three guidelines</p>
<p><strong>Filtering on the basis of view_kwargs or injecting something into view_kwargs</strong></p>
<p>This is the main point to discuss, starting from a simple scenario where we have to show different events list for different users. Before decoupling API server, we had two different routes, one served the public events listing on the basis of event identifier and other to show events to the event admins and managers, listing only their own events to their panel.</p>
<p>In API server there are no two different routes for this. We manage this with single route and served both cases using decorator. This below is the magic decorator function for this purpose</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">accessible_role_based_events</span><span class="p">(</span><span class="n">view</span><span class="p">,</span> <span class="n">view_args</span><span class="p">,</span> <span class="n">view_kwargs</span><span class="p">,</span> <span class="o">*</span><span class="n">args</span><span class="p">,</span> <span class="o">**</span><span class="n">kwargs</span><span class="p">):</span>
<span class="k">if</span> <span class="s">'POST'</span> <span class="ow">in</span> <span class="n">request</span><span class="o">.</span><span class="n">method</span> <span class="ow">or</span> <span class="s">'withRole'</span> <span class="ow">in</span> <span class="n">request</span><span class="o">.</span><span class="n">args</span><span class="p">:</span>
<span class="n">_jwt_required</span><span class="p">(</span><span class="n">app</span><span class="o">.</span><span class="n">config</span><span class="p">[</span><span class="s">'JWT_DEFAULT_REALM'</span><span class="p">])</span>
<span class="n">user</span> <span class="o">=</span> <span class="n">current_identity</span>
<span class="k">if</span> <span class="s">'GET'</span> <span class="ow">in</span> <span class="n">request</span><span class="o">.</span><span class="n">method</span> <span class="ow">and</span> <span class="n">user</span><span class="o">.</span><span class="n">is_staff</span><span class="p">:</span>
<span class="k">return</span> <span class="n">view</span><span class="p">(</span><span class="o">*</span><span class="n">view_args</span><span class="p">,</span> <span class="o">**</span><span class="n">view_kwargs</span><span class="p">)</span>
<span class="n">view_kwargs</span><span class="p">[</span><span class="s">'user_id'</span><span class="p">]</span> <span class="o">=</span> <span class="n">user</span><span class="o">.</span><span class="nb">id</span>
<span class="k">return</span> <span class="n">view</span><span class="p">(</span><span class="o">*</span><span class="n">view_args</span><span class="p">,</span> <span class="o">**</span><span class="n">view_kwargs</span><span class="p">)</span>
</code></pre></div></div>
<p>It works simply by looking for âwithRoleâ in requests and make a decision to include <code class="highlighter-rouge">user_id</code> into kwargs as per these rules</p>
<ol>
<li>If the request is POST then it has to be associated with some user so add the <code class="highlighter-rouge">user_id</code></li>
<li>If the request is GET and âwithRoleâ GET parameter is present in URL then yes add the <code class="highlighter-rouge">user_id</code>. This way user is asking to list the events in which I have some admin or manager role</li>
<li>If the request is GET and âwithRoleâ is defined but the logged in user is <code class="highlighter-rouge">admin</code> or <code class="highlighter-rouge">super_admin</code> then there is no need add <code class="highlighter-rouge">user_id</code> since staff can see all events in admin panel</li>
<li>The last one is GET and no âwithRoleâ parameter is defined therefore ignores and continues the same request to list all events.</li>
</ol>
<p>The next work is of <code class="highlighter-rouge">query</code> method of <code class="highlighter-rouge">EventList</code> Resource</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">if</span> <span class="n">view_kwargs</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s">'user_id'</span><span class="p">):</span>
<span class="k">if</span> <span class="s">'GET'</span> <span class="ow">in</span> <span class="n">request</span><span class="o">.</span><span class="n">method</span><span class="p">:</span>
<span class="n">query_</span> <span class="o">=</span> <span class="n">query_</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">Event</span><span class="o">.</span><span class="n">roles</span><span class="p">)</span><span class="o">.</span><span class="n">filter_by</span><span class="p">(</span><span class="n">user_id</span><span class="o">=</span><span class="n">view_kwargs</span><span class="p">[</span><span class="s">'user_id'</span><span class="p">])</span> \
<span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">UsersEventsRoles</span><span class="o">.</span><span class="n">role</span><span class="p">)</span><span class="o">.</span><span class="nb">filter</span><span class="p">(</span><span class="n">Role</span><span class="o">.</span><span class="n">name</span> <span class="o">!=</span> <span class="n">ATTENDEE</span><span class="p">)</span>
</code></pre></div></div>
<p>This query joins the UsersEventsRoles model whenever <code class="highlighter-rouge">user_id</code> is defined. Thus giving role-based events only.</p>
<p>The next interesting part is the Implementation of permission manager to ensure it doesnât break at any point. We will see it in next post.</p>poushOne of the interesting features of Python is the decorator. Decorators dynamically alter the functionality of a function, method, or class without having to directly use subclasses or change the source code of the function being decorated. Open Event API Server makes use of decorator in various ways. The ability to wrap a function and run the decorator(s) before executing that function solves various purpose in Python. Earlier before decoupling of Orga Server into API Server and Frontend, decorators were being used for routes, permissions, validations and more.Understanding {json:api} and itâs usage in Orga Server2017-06-27T01:31:11+02:002017-06-27T01:31:11+02:00/fossasia/oauth/2017/06/27/understanding-jsonapi-and-its-implementation<h2 id="what-is-an-api"><em>What is an API?</em></h2>
<p>API stands for âapplication programming interfaceâ. Put briefly, an API consists of a set of rules describing how one application can interact with another and the mechanisms that allow such interaction to happen.</p>
<h2 id="and-json-api"><em>and JSON API?</em></h2>
<p>JSON API is a specification for writing RESTFul APIs ( CRUD interfaces ). This specification basically sets the standard for a client to request the resources and how a server is supposed to response minimizing the redundancy and number of requests.</p>
<p>If we look at the general implementation of RESTful APIs, we see that we are working on creating every endpoint manually, there are no relations. Sometimes different endpoints are being created for some slightly different business logic than other.</p>
<h2 id="features"><em>Features</em></h2>
<p>Apart from CRUD interface, JSON-API-Spec provides</p>
<ul>
<li>Fetching Resources</li>
<li>Fetching Relationships</li>
<li>Inclusion of Related Resources</li>
<li>Sparse Fieldsets</li>
<li>Sorting</li>
<li>Pagination</li>
<li>Filtering</li>
</ul>
<p>And what we need at Open Event Orga Server?</p>
<ul>
<li>Proper relationship definitions</li>
<li>Sorting</li>
<li>Filtering</li>
<li>Pagination</li>
</ul>
<p>So JSON-API spec is a good choice for us at Orga Server since it solves our every basic need.</p>
<h2 id="overview-of-changes">Overview of Changes</h2>
<p>Firstly the main task was shifting to the library <a href="https://github.com/miLibris/flask-rest-jsonapi">flask-rest-jsonapi</a> because this library stands to our four needs in API.
The changes included:</p>
<ul>
<li>ensuring JSON-API spec in our requests and responses (although the most of the work is done by the library)</li>
<li>Reusing the current implementation of JWT authorization.</li>
<li>To locate the new API to <code class="highlighter-rouge">/v1</code>. Since Orga server is going to be API server with Open Event system following the API-centric approach, therefore, there is no need to have <code class="highlighter-rouge">/api/v1</code></li>
<li>Now out timestamps in response and request will be timezone aware thus following ISO 8601 with timezone information <code class="highlighter-rouge">(Eg. 2017-05-22T09:12:44+00:00)</code></li>
</ul>
<p><em>Following some basic rules like
A JSON object MUST be at the root of every JSON API request and response containing data. This object defines a documentâs âtop levelâ.
A document MUST contain at least one of the following top-level members:</em></p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>data: the documentâs âprimary dataâ
errors: an array of error objects
meta: a meta object that contains non-standard meta-information.
</code></pre></div></div>
<h4 id="media-type-to-use-applicationvndapijson">Media type to use: <code class="highlighter-rouge">application/vnd.api+json</code></h4>
<h2 id="an-example-of-new-api-server">An example of new API Server</h2>
<h4 id="resuest">Resuest</h4>
<pre><code class="language-JSON">POST /v1/users HTTP/1.1
Content-Type: application/json
POST /v1/users HTTP/1.1
Content-Type: application/vnd.api+json
Host: localhost:5000
Connection: close
User-Agent: Paw/3.1.1 (Macintosh; OS X/10.12.3) GCDHTTPRequest
Content-Length: 165
{
"data": {
"attributes": {
"name": "Open Event User",
"email": "example@example.com",
"password": "12345678"
},
"type": "user"
}
}
</code></pre>
<h4 id="response">Response</h4>
<pre><code class="language-JSON">HTTP/1.1 200 OK
Content-Type: application/vnd.api+json
{
"data": {
"attributes": {
"is-admin": false,
"last-name": null,
"instagram-url": null,
"is-super-admin": false,
"thumbnail-image-url": null,
"created-at": "2017-06-27T09:08:59.369531+00:00",
"last-accessed-at": null,
"email": "example@example.com",
"icon-image-url": null,
"contact": null,
"deleted-at": null,
"small-image-url": null,
"facebook-url": null,
"details": null,
"is-verified": false,
"first-name": null,
"avatar-url": null,
"twitter-url": null,
"google-plus-url": null
},
"type": "user",
"id": "2",
"links": {
"self": "/v1/users/2"
}
},
"jsonapi": {
"version": "1.0"
},
"links": {
"self": "/v1/users/2"
}
}
</code></pre>
<p>Looking at the example we can directly see how easy it is for us to manage different needs like pagination, relationships using JSON API.
Next steps in the implementation is Docs for APIs, permissions implementations to secure the endpoints and setting up unit testing of the endpoints which will be discussed in next posts.</p>
<p>Further read on JSON API Spec -> <a href="http://jsonapi.org/format/">http://jsonapi.org/format/</a></p>poushWhat is an API? API stands for âapplication programming interfaceâ. Put briefly, an API consists of a set of rules describing how one application can interact with another and the mechanisms that allow such interaction to happen.Using HTTMock to mock 3rd Party APIs2017-05-29T00:00:00+02:002017-05-29T00:00:00+02:00/2017/05/29/using-httmock-to-mock-3rd-party-apis<p>In the process of implementing the connected social media in API server. There was a situation where we need to mock the 3rd party API services like Google OAuth, Facebook Graph API.</p>
<p>The idea was simple, as suggested by <a href="https://github.com/hongquan">@hongquan</a>, when <code class="highlighter-rouge">requests</code> make a <strong>HTTP</strong> request to <em>https://api.google.com/profile</em>, for example, <a href="https://github.com/patrys/httmock">httmock</a> will</p>
<ul>
<li>stand in the middle,</li>
<li>stop request from going to the Internet,</li>
<li>and returns a JSON response as if the response is from Google.</li>
</ul>
<p>The content of this response is written by us in the test case. We have to read Google documentation to write a correct fake response.</p>
<p><strong><em>Library used</em></strong></p>
<p>For this purpose, I used the <a href="https://github.com/patrys/httmock">httmock</a> library for Python.</p>
<h3 id="steps-to-follow">Steps to follow</h3>
<ol>
<li>Look for response object on two requests (OAuth and profile details).</li>
<li>Create the dummy response using the sample response object.</li>
<li>Creating endpoints using the httpmock library.</li>
<li>During test run, calling the specific method <code class="highlighter-rouge">with HTTMock</code></li>
</ol>
<p>Sample object of OAuth Response from Google is:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> {
"access_token":"2YotnFZFEjr1zCsicMWpAA",
"token_type":"Bearer",
"expires_in":3600,
"refresh_token":"tGzv3JOkF0XG5Qx2TlKWIA",
"example_parameter":"example_value"
}
</code></pre></div></div>
<p>and from the sample object of Google Profile API we needed the link of profile for our API-server:</p>
<pre><code class="language-JSON">{'link':'http://google.com/some_id'}
</code></pre>
<p><strong>Creating the dummy response</strong></p>
<p>Creating dummy response was easy. All I had to do is provide proper header and content in response and use <code class="highlighter-rouge">urlmatch</code> decorator</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="c1"># response for getting userinfo from google
</span><span class="o">@</span><span class="n">urlmatch</span><span class="p">(</span><span class="n">netloc</span><span class="o">=</span><span class="s">'https://www.googleapis.com/userinfo/v2/me'</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">google_profile_mock</span><span class="p">(</span><span class="n">url</span><span class="p">,</span> <span class="n">request</span><span class="p">):</span>
<span class="n">headers</span> <span class="o">=</span> <span class="p">{</span><span class="s">'content-type'</span><span class="p">:</span> <span class="s">'application/json'</span><span class="p">}</span>
<span class="n">content</span> <span class="o">=</span> <span class="p">{</span><span class="s">'link'</span><span class="p">:</span><span class="s">'http://google.com/some_id'</span><span class="p">}</span>
<span class="k">return</span> <span class="n">response</span><span class="p">(</span><span class="mi">200</span><span class="p">,</span> <span class="n">content</span><span class="p">,</span> <span class="n">headers</span><span class="p">,</span> <span class="bp">None</span><span class="p">,</span> <span class="mi">5</span><span class="p">,</span> <span class="n">request</span><span class="p">)</span>
<span class="o">@</span><span class="n">urlmatch</span><span class="p">(</span><span class="n">netloc</span><span class="o">=</span><span class="s">r'(.*\.)?google\.com$'</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">google_auth_mock</span><span class="p">(</span><span class="n">url</span><span class="p">,</span> <span class="n">request</span><span class="p">):</span>
<span class="n">headers</span> <span class="o">=</span> <span class="p">{</span><span class="s">'content-type'</span><span class="p">:</span> <span class="s">'application/json'</span><span class="p">}</span>
<span class="n">content</span> <span class="o">=</span> <span class="p">{</span>
<span class="s">"access_token"</span><span class="p">:</span><span class="s">"2YotnFZFEjr1zCsicMWpAA"</span><span class="p">,</span>
<span class="s">"token_type"</span><span class="p">:</span><span class="s">"Bearer"</span><span class="p">,</span>
<span class="s">"expires_in"</span><span class="p">:</span><span class="mi">3600</span><span class="p">,</span>
<span class="s">"refresh_token"</span><span class="p">:</span><span class="s">"tGzv3JOkF0XG5Qx2TlKWIA"</span><span class="p">,</span>
<span class="s">"example_parameter"</span><span class="p">:</span><span class="s">"example_value"</span>
<span class="p">}</span>
<span class="k">return</span> <span class="n">response</span><span class="p">(</span><span class="mi">200</span><span class="p">,</span> <span class="n">content</span><span class="p">,</span> <span class="n">headers</span><span class="p">,</span> <span class="bp">None</span><span class="p">,</span> <span class="mi">5</span><span class="p">,</span> <span class="n">request</span><span class="p">)</span>
</code></pre></div></div>
<p>So now we have the end points to mock the response. All we need to do is to use HTTMock inside the test case.</p>
<p>To use this setup all we need to do is:</p>
<div class="language-python highlighter-rouge"><div class="highlight"><pre class="highlight"><code> <span class="k">with</span> <span class="n">HTTMock</span><span class="p">(</span><span class="n">google_auth_mock</span><span class="p">,</span> <span class="n">google_profile_mock</span><span class="p">):</span>
<span class="bp">self</span><span class="o">.</span><span class="n">assertTrue</span><span class="p">(</span><span class="s">'Open Event'</span> <span class="ow">in</span> <span class="bp">self</span><span class="o">.</span><span class="n">app</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s">'/gCallback/?state=dummy_state&code=dummy_code'</span><span class="p">,</span>
<span class="n">follow_redirects</span><span class="o">=</span><span class="bp">True</span><span class="p">)</span><span class="o">.</span><span class="n">data</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">assertEqual</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">app</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s">'/gCallback/?state=dummy_state&code=dummy_code'</span><span class="p">)</span><span class="o">.</span><span class="n">status_code</span><span class="p">,</span> <span class="mi">302</span><span class="p">)</span>
<span class="bp">self</span><span class="o">.</span><span class="n">assertEqual</span><span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">app</span><span class="o">.</span><span class="n">get</span><span class="p">(</span><span class="s">'/gCallback/?state=dummy_state&code=dummy_code'</span><span class="p">)</span><span class="o">.</span><span class="n">status_code</span><span class="p">,</span> <span class="mi">302</span><span class="p">)</span>
</code></pre></div></div>
<p>And we were able to mock the Google APIs in our test case. Complete implementation in Fossasia API-Server can be seen <a href="https://github.com/fossasia/open-event-orga-server/pull/3629/files">Here</a></p>poushIn the process of implementing the connected social media in API server. There was a situation where we need to mock the 3rd party API services like Google OAuth, Facebook Graph API.Connecting Social Apps with Orga API Server2017-05-23T04:11:24+02:002017-05-23T04:11:24+02:00/fossasia/oauth/2017/05/23/connecting-social-apps-with-orga-api-server<h2 id="addition-of-a-feature-to-orga-api-server">Addition of a feature to Orga API Server</h2>
<h3 id="ability-to-connect-your-social-apps-with-orga-server"><em>Ability to connect your social apps with orga server</em></h3>
<p><strong><em>Whatâs going to add?</em></strong></p>
<p>A feature which will allow us to provide Organizer options to connect with their social media audience directly with API server and Users to share their experience for different events on their social media platforms.</p>
<p><strong><em>Platforms added</em></strong></p>
<ul>
<li><strong>F</strong>aceBook</li>
<li><strong>T</strong>witter</li>
<li><strong>I</strong>nstagram</li>
<li><strong>G</strong>oogle+</li>
</ul>
<h3 id="the-auth-tool---oauth-20">The Auth Tool - OAuth 2.0</h3>
<p>Without going on introductory introduction to Oauth2.0 Letâs focus on its implementation in Orga API Server</p>
<p><strong>OAuth Roles</strong></p>
<p><em>OAuth defines four roles:</em></p>
<ul>
<li>Resource Owner</li>
<li>Client</li>
<li>Resource Server</li>
<li>Authorization Server</li>
</ul>
<p>Letâs look at the responsibility of these roles when you connect your social apps with API server</p>
<p><strong>Resource Owner: User/Organizer ( You )</strong></p>
<p>You as User or Organizer connection your accounts with API server are the resource owners who authorize API server to access your account. During authorization, you provide us access to read your account details like Name, Email, Profile photo, etc.</p>
<p><strong>Resource / Authorization Server: Social Apps</strong></p>
<p>The resource server here is your social platforms/apps where you have your account registered. These Apps provide us limited access to fetch details of your account once you authorize our application to do so.
They make sure the token we provide match with the authorization provided before through your account.</p>
<p><strong>Client: Orga API Server</strong></p>
<p>Orga API Server acts as the client to access your account details. Before it may do so, it must be authorized by the user, and the authorization must be validated by the API.</p>
<h3 id="the-process-to-add">The process to add</h3>
<p>A simple work plan to follow:</p>
<ol>
<li>Understanding how OAuth is implemented.</li>
<li>Test OAuth implementation on all 4 social medias.</li>
<li>After Necessary correction. Make sure we have all views(routes) to connect these 4 social medias.</li>
<li>Implementing the same feature on the template file.</li>
<li>Make sure these connect buttons are shown only when Admin has registered its client credentials in Settings.</li>
<li>Creating a view to unlink your social media account.</li>
</ol>
<p><strong>Understanding how OAuth is implemented.</strong></p>
<p>Current Implementation of OAuth is very simple and interesting on API server. We have OAuth helper classes which provide all necessary endpoints and different methods to get the job done.
<img src="/Screen%20Shot%202017-05-23%20at%205.17.52%20PM.png" alt="Oauth" /></p>
<p><strong>Test OAuth implementation on all 4 social medias.</strong></p>
<p>Now we can work on testing on the callbacks of all 4 social apps. We have callback defined in <code class="highlighter-rouge">views/util_routes.py</code>
For this, I picked up the auth OAuth URLs and called them directly on my browsers and testing their callback. Now on callback, those methods required some change to save user data on database thus connecting their accounts with API server. This lead to changes in <code class="highlighter-rouge">update_user_details</code> and on callback methods.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>def update_user_details(first_name=None,
last_name=None,
facebook_link=None,
twitter_link=None,
file_url=None,
instagram=None,
google=None):
</code></pre></div></div>
<p><strong>Make sure we have all views(routes) to connect these 4 social medias</strong></p>
<p>This has to be done on <code class="highlighter-rouge">views/users/profile.py</code>
Addition of one method</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>@profile.route('/google_connect/', methods=('GET', 'POST'))
def google_connect():
....
....
return redirect(gp_auth_url)
</code></pre></div></div>
<p>and testing, correction on other 3 methods</p>
<p><strong>Implementing the same feature on template file.</strong></p>
<p>Updating <code class="highlighter-rouge">gentelella/users/settings/pages/applications.html</code> to add changes required to add this feature. This included ability to show URLs of connected accounts and functioning connect and disconnect button</p>
<p><strong>Make sure these connect buttons are shown only when Admin has registered its client credentials in Settings.</strong></p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> fb = get_settings()['fb_client_id'] != None and get_settings()['fb_client_secret'] != None
....
....
...
</code></pre></div></div>
<p>The addition of such snippet provides data to the template to decide whether to show those fields or not. It will not make any sense if there is no application created to connect those accounts by Admin.</p>
<p><strong>Creating a view to unlink your social media account.</strong></p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>utils_routes.route('/unlink-social/<social>')
def unlink_social(social):
if login.current_user is not None and login.current_user.is_authenticated:
...
...
</code></pre></div></div>
<p>A method is created to unlink the connected accounts so that users can anytime disconnect their accounts from API server.</p>
<p><strong>Where to connect?</strong></p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>Settings > Applications
</code></pre></div></div>
<p><em>How it Works</em>
<img src="https://www.dropbox.com/s/ra0fi45mo7i6jby/ScreenFlow.gif?dl=1" alt="Download" /></p>poushAddition of a feature to Orga API Server Ability to connect your social apps with orga serverUsing Cloud storage for event exports2017-05-09T09:56:44+02:002017-05-09T09:56:44+02:00/fossasia/2017/05/09/using-cloud-storage-for-event-exports<p>Open-event orga server provides ability to organizer to create complete export of the event they created. Currently when an organizer triggers the export in orga server, A celery job is set to complete the export task resulting asynchronous completion of the job. Organizer gets the download button enabled once export is ready.</p>
<p>Till now the main issue was related with storage of those export zip files. All exported zip files were stored directly in local storage and that even not by using storage module created under orga server.</p>
<p><img src="/Screen%20Shot%202017-05-10%20at%207.11.36%20PM.png" alt="local storage path" /></p>
<p>On a mission to solve this, I made three simple steps that I followed to solve this issue.</p>
<p>These three steps were:</p>
<blockquote>
<ol>
<li>Wait for <strong>shutil.make_archive</strong> to complete archive and store it in local storage.</li>
<li>Copy the created archive to storage ( specified by user )</li>
<li>Delete local archive created.</li>
</ol>
</blockquote>
<p>The easiest part here was to make these files upload to different storage ( s3, gs, local) as we already have <code class="highlighter-rouge">storage</code> helper</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>def upload(uploaded_file, key, **kwargs):
"""
Upload handler
"""
</code></pre></div></div>
<p>The most important logic of this issue resides to this code snippet.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> dir_path = dir_path + ".zip"
storage_path = UPLOAD_PATHS['exports']['zip'].format(
event_id = event_id
)
uploaded_file = UploadedFile(dir_path, dir_path.rsplit('/', 1)[1])
storage_url = upload(uploaded_file, storage_path)
if get_settings()['storage_place'] != "s3" or get_settings()['storage_place'] != 'gs':
storage_url = app.config['BASE_DIR'] + storage_url.replace("/serve_","/")
return storage_url
</code></pre></div></div>
<p>From above snippet it is clear that we are extending the process of creating zip. Once the zip is created we will make storage path for cloud storage and upload it. Only one thing will take time to understand here is the last second and third line of above snippet.</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code>if get_settings()['storage_place'] != "s3" or get_settings()['storage_place'] != 'gs':
storage_url = app.config['BASE_DIR'] + storage_url.replace("/serve_","/")
</code></pre></div></div>
<p>Initial the plan was simple to serve the files through <strong><code class="highlighter-rouge">serve_static</code></strong> but then the test cases were expecting a file at this location thus I had to remove âserve_â part for local storage and then it works fine on those three steps.</p>
<p>Next thing on this storage process need to be discussed is feature to delete old exports. I believe one reason to keep them would be a old backup of your event will be always there with us at our cloud storage.</p>poushOpen-event orga server provides ability to organizer to create complete export of the event they created. Currently when an organizer triggers the export in orga server, A celery job is set to complete the export task resulting asynchronous completion of the job. Organizer gets the download button enabled once export is ready.Different Types of SQL Commands2017-04-25T09:56:44+02:002017-04-25T09:56:44+02:00/general/windows/2017/04/25/different-types-of-sql<p>Structured Query Language (aka âSQLâ) is created for managing RDBMS ( Relational Database Management Systems )</p>
<p>From Wikipedia,</p>
<blockquote>
<p>Originally based upon relational algebra and tuple relational calculus, SQL consists of a data definition language, data manipulation language, and data control language. The scope of SQL includes data insert, query, update and delete, schema creation and modification, and data access control. Although SQL is often described as, and to a great extent is, a declarative language (4GL), it also includes procedural elements.</p>
</blockquote>
<p>Looks Good!
In simple terms, we are using SQL to operate our RDBMS and this language has seriously easy syntax.</p>
<p>Now there are different types of SQL commands. ( and yeah! We run commands to operate our RDBMS in SQL)</p>
<ol>
<li>DDL - Data Definition Language</li>
<li>DML - Data Manipulation Language</li>
<li>DCL - Data Control Language</li>
</ol>
<p>Till my viva in my college, I wasnât aware of any third type :D</p>
<h2 id="ddl">DDL</h2>
<p>DDL provides the standard structure of commands that defines the structure of database objects like tables, indexes. Some of the operations/commands are</p>
<ul>
<li>CREATE TABLE</li>
<li>ALTER TABLE</li>
<li>DROP TABLE</li>
<li>CREATE INDEX</li>
<li>ALTER INDEX</li>
<li>DROP INDEX</li>
<li>CREATE VIEW</li>
<li>DROP VIEW</li>
</ul>
<h2 id="dml">DML</h2>
<p>These commands are used for managing data within schema objects. Some examples:</p>
<ul>
<li>INSERT - insert data into a table</li>
<li>UPDATE - updates existing data within a table</li>
<li>DELETE - deletes all records from a table, the space for the records remain</li>
</ul>
<h2 id="dcl">DCL</h2>
<p>These commands manage the access to your database. Generally applied to database users so that you can control the access of other users to your database.</p>
<ul>
<li>ALTER PASSWORD</li>
<li>GRANT</li>
<li>REVOKE</li>
<li>CREATE SYNONYM</li>
</ul>
<p>Thatâs all summarized for three different types of SQL commands.</p>
<h2 id="wait">Wait!</h2>
<p>You see SELECT command anywhere? If you are familiar with SQL then you might know about most used command, The âSELECTâ command and if you see the lists above you will not find it in any category. Well there is another category called</p>
<h3 id="dql---data-query-language">DQL - Data Query language</h3>
<p>Comprises only one command <strong><em>SELECT</em></strong> but is most used and most concentrated for SQL users. With different options and clauses, it is used to make queries (inquiries) to the database.</p>
<p>Take the simple example, when you open a website page then the SELECT queries are made to get user data, website configurations, data to shown on web page. Complex and large data require joins and multiple queries.
And yes number of queries can be reduced by different techniques but we will discuss it later :)</p>poushStructured Query Language (aka âSQLâ) is created for managing RDBMS ( Relational Database Management Systems )