Category Archives: Python

Sentiment analysis using Python and TextBlob

Let’s create a really simple FastAPI with, create yourself our app file (app.py) and requirements file (requirements.txt).

We’re going to use TextBlob to process our text.

In the requirements.txt add the following

fastapi
uvicorn
textblob

In the app.py add the following imports

from fastapi import FastAPI
from textblob import TextBlob

Now let’s create the FastAPI and a POST endpoint named sentiment, the code should look like this

@app.post("/sentiment")
def analyze_sentiment(payload: dict):
    text = payload["text"]
    blob = TextBlob(text)
    polarity = blob.sentiment.polarity
    subjectivity = blob.sentiment.subjectivity
    return {
        "polarity": polarity,
        "subjectivity": subjectivity
    }

Don’t forget to run pip install

pip install -r requirements.txt

or if you’re using PyCharm, let this install the dependencies.

Run the app using

uvicorn app:app --reload

Note: as we’re using FastAPI we can access the OpenAPI interface using http://localhost:8000/docs

Now from curl run

curl -X POST http://localhost:8000/sentiment -H "Content-Type: application/json" -d '{"text": "I absolutely love this!"}'

and you’ve see a result along the following lines

{"polarity":0.625,"subjectivity":0.6}

Polarity is within the range [-1.0, 1.0], where -1.0 is a very negative sentiment, 0, neutral sentiment and 1.0 very positive sentiment. Subjectivity is in the range [0.0. 1.0] where 0.0 is very objective (i.e. facts or neutral statements) and 1.0 is very subjective (i.e. opinions, feelings or personal judgement).

A simple web API in various languages and deployable to Kubernetes (Python)

Continuing this short series of writing a simple echo service web API along with the docker and k8s requirements, we’re now going to turn our attention to a Python implementation.

Implementation

I’m using JetBrains PyCharm for this project, so I created a project named echo_service.

Next, add the file app.py with the following code

from flask import Flask, request

app = Flask(__name__)

@app.route('/echo')
def echo():
    text = request.args.get('text', '')
    return f"Python Echo: {text}", 200

@app.route('/livez')
def livez():
    return "OK", 200

@app.route('/readyz')
def readyz():
    return "Ready", 200

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=8080)

Add a requirements.txt file with the following

flask
gunicorn

Don’t forget to install the packages via the IDE.

Dockerfile

Next up we need to create our Dockerfile

# Use a lightweight Python base
FROM python:3.11-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY app.py .

CMD ["gunicorn", "-w", "2", "-b", "0.0.0.0:8080", "app:app"]

Note: we’ll be using gunicorn instead of the development server.

Note: In Linux port 80 might be locked down, hence we use port 8080 by default.

To build this, run

docker build -t putridparrot.echo_service:v1 .

Don’t forget to change the name to your preferred name.

and to test this, run

docker run -p 8080:8080 putridparrot.echo_service:v1

Kubernetes

If all wen well we’ve not tested our application and see it working from a docker image, so now we need to create the deployment etc. for Kubernete’s. Let’s assume you’ve pushed you image to Docker or another container registry such as Azure – I’m call my container registry putridparrotreg.

I’m also not going to use helm at this point as I just want a (relatively) simple yaml file to run from kubectl, so create a deployment.yaml file, we’ll store all the configurations, deployment, service and ingress in this one file jus for simplicity.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: echo
  namespace: dev
  labels:
    app: echo
spec:
  replicas: 1
  selector:
    matchLabels:
      app: echo
  template:
    metadata:
      labels:
        app: echo
    spec:
      containers:
      - name: echo
        image: putridparrotreg/putridparrot.echo_service:v1
        ports:
        - containerPort: 8080
        resources:
          requests:
            memory: "100Mi"
            cpu: "100m"
          limits:
            memory: "200Mi"
            cpu: "200m"
        livenessProbe:
          httpGet:
            path: /livez
            port: 8080
          initialDelaySeconds: 30
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /readyz
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 5

---
apiVersion: v1
kind: Service
metadata:
  name: echo_service
  namespace: dev
  labels:
    app: echo
spec:
  type: ClusterIP
  selector:
    app: echo 
  ports:
  - name: http
    port: 80
    targetPort: 8080
    protocol: TCP
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: echo-ingress
  namespace: dev
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  rules:
  - host: mydomain.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: echo_service
            port:
              number: 80

Don’t forget to change the “host” and image to suit, also this assume you created a namespace “dev” for your app. See Creating a local container registry for information on setting up your own container registry.

Python REST service using Flask

Wanting to create a Python based REST service? Let’s use Flask and see what we can do.

Start off by installing Flask

pip install Flask

Now let’s write some code

from flask import Flask
app = Flask(__name__)

@app.route("/")
def hello():
    return "Hello World!"

app.run()

This is nice and simple and pretty obvious how things work. As you can see we create the Flash application and in this sample run it (although on the Flask website they often show running the Flask Run from the Python command line).

Each “route” is defined and mapped directly to a function. In this example the root of the URL is mapped to the hello function.

The route can also contain variables which are show within <>, for example

@app.route('/<name>')
def hello(name):
    return f"Hello {name}"

We can also define converters (similar to declaring the type of the variable) for the variables, so for example if we need to access something by an integer id, the we might have

@app.route('/employee/<int:id>')
def employee(id):
    return f"Employee {id}"

By default the converter used (as you’d probably expect) is string, but Flask also supports int, float, path, any and uuid.

By default each route is using the HTTP method GET, but we can also define the supported methods for each route, i.e.

@app.route('/login', methods=['GET', 'POST'])
def login():
    if request.method == 'POST':
        do_the_login()
    else:
        show_the_login_form()

References

Flask is Fun

Python REST service using web.py

First off, run either of the following commands from PyCharm’s Terminal window (or the command line, ofcourse).

pip install web.py
# or
easy_install web.py

From PyCharm you can also go to File | Settings, then locate Project Interpreter, for your project. Here you can press the + button, search for web.py then press the Install Package button.

If we take the Getting Started code from the web.py website and place into a .py file

import web

urls = (
    '/(.*)', 'hello'
)
app = web.application(urls, globals())


class hello:
    def GET(self, name):
        if not name:
            name = 'World'
        return 'Hello, ' + name + '!'


if __name__ == "__main__":
    app.run()

Run this and you should see the output http://0.0.0.0:8080. Now, from your preferred web browser type http://localhost:8080. The default result will be “Hello hello!” if you now type http://localhost:8080/PutridParrot you’ll see “Hello PutridParrot!”.

So what’s happening…?

urls = (
    '/(.*)', 'hello'
)
app = web.application(urls, globals())

The web.application creates a web applications with the supplied url’s and the classes that each url maps to. In this instance all url’s map to the hello class. Now the hello class is

class hello:
    def GET(self, name):
        if not name:
            name = 'World'
        return 'Hello, ' + name + '!'

So this class takes all url’s and defines a function for each HTTP method accepts, the case maps to that sent via the browser, hence is uppercase GET (if you make a mistake here you’ll see a Method Not Allowed error from your REST service).

Our GET method takes an argument, in the case of http://localhost:8080/PutridParrot this argument is Putridparrot. If no argument is supplied the code sets name to ‘World’. You get the idea.

Finally we need to run the service application, hence the code

if __name__ == "__main__":
    app.run()

Templating

So this is all well and good, but the output is plain text, what if we are serving web pages, it’d be a lot better to be able to write HTML and embed our Python code in it, like PHP, ASP etc.

web.py includes a templating capability, so if we create a folder for our templates (let’s call it templates) off of our service source. Then add an HTML file, let’s call it index.html, now remove the HTML from this and replace with

$def with(name)

$if name:
   Hello <em>$name</em>
$else:
   Hello <em>World</em>

We’ve got the first line as $def with(name) which basically states that our template is called with the variable name which we then use within the rest of the template. The $ obviously preceeds our Python statements, but other than that it’s probably pretty obvious what’s going on here and fairly standard looking for Python.

Now change our python source to look like this

import web

urls = (
    '/(.*)', 'index'
)

render = web.template.render('templates/')

class index:
    def GET(self, name):
        return render.index(name)


if __name__ == "__main__":
    app = web.application(urls, globals())
    app.run()

We need to tell web.py where our templates reside, hence the line

render = web.template.render('templates/')

Now I’ve changed the class etc. to be our index class, this wasn’t necessary as render.index maps to our HTML file name, but I figured I’d try and make things more like a real world application.

Python REST service using Bottle

From the command line/terminal run

pip install bottle

Bottle supports a style of defining routes to functions using decorators, for example

from bottle import route, run

@route('/<name>')
def index(name):
    return f"Hello {name}"

run(host='localhost', port=8080)

HTTP Method(s)

By default a @route will respond to a GET HTTP method, but we can change to POST or be more explicit using the following

@route('/<name>', method='GET')
# or
@get('/<name>')
# or POST
@route('/<name>', method='POST')
# or
@post('/<name>')

We can combine the methods using the following

@route('/<name>', method=['GET', 'POST'])

and then within the actual function we can use the request class like this

@route('/<name>', method=['GET', 'POST'])
def index(name):
    if request.method == 'POST':
       return f"POST {name}"
    elif request.method == 'GET':
       return f"GET {name}"

    return "Unexpected"

We can also declare multiple routes to a function like this

@route('/', method=['GET', 'POST'])
@route('/<name>', method=['GET', 'POST'])
def index(name = None):
    if request.method == 'POST':
       return f"POST {name}"
    else request.method == 'GET':
       return f"GET {name}"

    return "Unexpected"

How about some JSON

Obviously JSON is the current “in vogue” format for transferring data. To return JSON instead of a simple string we can use jsonpickle, start off by installing the library using

pip install jsonpickle

We’ll add an example class for our result

class Person:
    def __init__(self):
        self.firstName = "Eddie"
        self.lastName = "Van Halen"

and our code (and additional imports) looks like this

import jsonpickle

@get('/<name>')
def index(name):
    person = Person()
    return jsonpickle.encode(person, unpicklable=False)

The unpicklable=False turns off the pickle additional information to our JSON.

Errors

We can also supply web pages, data etc. based upon HTTP errors, such as 404 errors. Assuming we changed out @get (above) to

@get(‘/name/<name>’)

now URL’s such as http://localhost:8080/PutridParrot will fail to find a matching route and if we supply an error handler for a 404 then we can return some other data, i.e.

@error(404)
def error404(error):
    return "Got a 404!"

References

Bottle: Python Web Framework

Mocks and Python

In my post Unit testing in Python I covered the basics of writing unit tests within Python.

Testing also sometimes involves writing stubs and/or using mock objects.

As Python is a dynamic language which supports duck typing, we can fairly easily create stub objects, but a mocking framework can make things even simpler. However a good mock framework also gives us the ability to check whether a method was called and assert that it was not called, or was called the correct number of times.

Using MagicMock

MagicMock is a class within the unittest.mock module used to patch methods. Patching being the process of swapping out methods with our mock methods.

Here’s a Calculator class with an add method

class Calculator:
    def add(self, a, b):
        return a + b

Let’s assume we want to mock/patch the add method and ensure it’s called a certain number of times

Note: this is a contrived example as we are explicitly calling the add method, thus know how many times it’s called, but it’s a simple enough example to see how things fit together.

from unittest.mock import MagicMock 

class CalculatorTests(TestCase):

    @classmethod
    def setUpClass(cls):
        cls.c = Calculator()

    def test_add(self):
        self.c.add = MagicMock(name='add')
        self.c.add.return_value = 5

        self.assertEqual(5, self.c.add(3, 2))
        self.c.add.assert_called_once_with(3, 2)

In the above we’re using assertEqual just to demonstrate the use of the return_value. The call to the add method within the assertEqual demonstrates how we still call our patched method in the same way and then finally we use assert_called_once_with(3, 2) to assert that the add method was call once and with the expected parameters.

References

unittest.mock — mock object library

Variable arguments in Python

When looking at some built-in functions within builtins.py, I came across the strange syntax

def ascii(*args, **kwargs): 

The *args and **kwargs has a special meaning within Python. The *args will give all function parameters as a tuple and the **kwargs will give a dictionary of key/values.

Note: convention seems to suggest the parameter names args and kwargs, but these are not enforced naming conventions.

Let’s see these two (separately) in action

def show_args(*args):
    for a in args:
        print(a)


def show_key_values(**kwargs):
    for a in kwargs:
        print(a, kwargs[a])


show_args("Hello", "World", 123)
show_key_values(first="Hello", second="World", third=123)

As you can see, in the show_args function we will list each argument out, hence the output to stdout will be

Hello
World
123

The show_key_values expects key/value input, hence the field/property name followed by = and then the value. This will output

first Hello
second World
third 123

In both cases we can also pass no arguments to these methods and nothing will be output, but it’s also not unusual (as can be seen by the ascii function at the start of this post) for the two pieces of syntax to be combined, i.e.

def show_args_key_values(*args, **kwargs):
    for a in args:
        print(a)

    for a in kwargs:
        print(a, kwargs[a])

and now this can be called in the following ways

show_args_key_values("Hello", "World", 123, first="Hello Again")

As you can probably tell from this syntax, you will need to list the list of arguments before the key/value list of arguments, but you can write the following

show_args_key_values("Hello")
show_args_key_values(first="World")

and therefore handle either a list of key/value set of input.

Unit testing in Python

As Python is a dynamic language, unit testing is not only important for verifying code works as expected, but also needs to cover situations that statically typed languages get for free. So a high level of test coverage is useful to ensure event basic methods are working and even named as expected.

Let’s create a simply Calculator class which we will then write unit tests for (this is saved in a file named calc.py)

class Calculator:
    def add(self, a, b):
        return a + b

We can then create a file test_calculator.py and within this we have the following

from unittest import TestCase
from calc import Calculator


class TestCalculator(TestCase):
    def test_add(self):
        c = Calculator()
        self.assertEqual(5, c.add(3, 2))

As you can see, we need to derive our test class from the TestCase class and test methods should use the naming convention test_ at the start of the test method’s name.

Note: the class name does not require the Test prefix/naming convention.

Using PyCharm we need to create a new configuration, from the Python tests configuration section. Select nosetests and then name the configuration, something like Calculator Tests.

Now we can run the tests. If you find nosetest is not installed you can run the Python Console from PyCharm and run the command

pip install nose

setUp/tearDown

Like most unit testing frameworks, the TestCase class can use a setUp and/or tearDown method for setting up the test case context and cleaning up after each test method is run.

For example

class CalculatorTests(TestCase):

    def setUp(self):
        self.c = Calculator()

    def test_add(self):
        self.assertEqual(5, self.c.add(3, 2))

    def test_subtract(self):
        self.assertEqual(2, self.c.subtract(7, 5))

setUpClass/tearDownClass

TestCase also includes setUpClass and tearDownClass for setting context for all test methods within a TestCase.

For example, here’s the previous TestCase but using the setUpClass to create the calculator for all test methods

class CalculatorTests(TestCase):

    @classmethod
    def setUpClass(cls):
        cls.c = Calculator()

    def test_add(self):
        self.assertEqual(5, self.c.add(3, 2))

    def test_subtract(self):
        self.assertEqual(2, self.c.subtract(7, 5))

Note: the @classmethod is required in this example code or the setUpClass is not treated as a class method as is expected.

References

unittest — Unit testing framework
Creating and running a Python unit test

Python exception handling

Python includes a bunch of built-in exception classes and we can easily create our own by creating a class which derives from Exception.

For example, here’s a MethodNotImplementedException type

class MethodNotImplementedException(Exception):
    def __init__(self, *args, **kwargs):
        pass

    def __str__(self):
        return "Method not implemented"

In this example I’ve overridden the __str__ method to return a simple string when we print the exception to stdout.

We need not create specific exception types but obviously this allows us to also catch specific exception types.

Here’s an example of some code that raises (throws) an exception and the code to catch and handle an exception

class Animal:
    def name(self):
        raise MethodNotImplementedException()


a = Animal()
try:
    a.name()
except MethodNotImplementedException as e:
    print(e)
except Exception as e:
    raise
finally:
    print("finally called")

We create a try block and exceptions are caught using the except statement. As can be seen we can catch and filter each exception type we want to handle and include a finally block as required.

To rethrow and exception we simple call the raise method.

Python classes

In my previous post I looked at some of the basics of Python. In this post I want to dig further into the syntax etc. for classes within Python.

Class naming conventions

We prefix a class name with the class keyword. Python naming convention suggests the class name should be Pascal case, i.e. ClassName.

The method naming convention, by default, should be all lowercase with underscores (snake case) to separate words, i.e. method_name.

Private/protected methods and instance variables should have an underscore prefix, i.e. _private_name.

Defining a class

The special method name __init__ can be thought of a little like a constructor, however we cannot overload it, but we can pass arguments into it

class Animal:
    def __init__(self, name):
        self.name = name

Note: Whilst it might appear possible to have multiple __init__ methods, it’s the last one declared which seems to be the one available to calling code.

We terminate the line that the class keyword is on with the colon and then methods or variables start on a new line and indented with a tab.

The self parameter passed to __init__ or any other class method is passed implicitly, i.e. we do not supply the self, Python does that for us.

Inheritance

Python classes support inheritance (even multiple inheritance) by declaring the base classes in a comma separated list within parenthesis, i.e.

class Animal:
    def __init__(self, name):
        self.name = name


class Mammal:
    pass


class Dog(Animal, Mammal):
    def __init__(self):
        super().__init__("Dog")

In this example you can see how we can call the base class’ __init__ method. The Dog class derives from both Animal and Mammal classes (in this instance the Mammal class has no methods etc. hence uses pass statement to create the empty class).

Calling base methods in Multiple Inheritance

Multiple inheritance always has the issue around what base method to call in situations where there’s multiple methods of the same name, for example if we have the following

class A:
    def name(self):
        print("A")

class B:
    def name(self):
        print("B")


class C(A, B):
    pass

c = C()
c.name()

then what base class method is actually called via c.name().

A quick note: In the above as we’re missing the method name() on the class C, this can be seen as the following

class C(A, B):
    def name(self):
        super(C, self).name()

The order of resolving the methods to be called is the Method Resolution Order (MRO).

For our simple example, the first name() method located depends upon the order of inheritance, i.e. in the example above A’s name() is called, switching the code for C to

class C(B, A):
    pass

results in B’s name method being called.

To ensure we are explicit about which code is called, it’s best to override the name method in the subclass like this

class C(A, B):
    def name(self):
        B.name(self)

and obviously this ensures B’s name is called.

Methods

We declare methods within a class using the keyword def and all methods are virtual, i.e. can be overridden in a subclass. Here’s a basic example over method overriding

class Animal:
    def name(self):
        pass


class Dog(Animal):
    def name(self):
        return "Dog"

In this example we’re using the Animal a little like an abstract class and then implementing the name method in derived classes.

Member variables/fields

Let’s rewrite the previous code to now use a member variable in the base class to store the name of the Animal.

class Animal:
    _name = ""

    def name(self):
        return self._name


class Dog(Animal):
    def __init__(self):
        self._name = "Dog"

In this case the prefixed _ denotes a protected/private variable. In most OO languages this encapsulates the member variable in such a way as to ensure it’s no accessible outside of the base and derived classes, however in Python it’s still available, and thus this will return the value stored within the _name variable. In PyCharm we get a hint that we’re accessing a protected member, but ultimately it’s still accessible, as per the example below

a = Dog()
print(a._name)

Empty classes

In some cases we might want to create an empty class type, such as

class Duck:
    pass

If we need to then create the equivalent of a struct at runtime, then we can simply dynamically “add” variables like this

d = Duck()
d.name = "Duck"

this leads us onto duck typing…

Duck typing

Duck typing allows us to declare Python type for use in other types that simple expect certain methods/variables to exist, for example

class Car:
    def __init__(self, engine):
        self.engine = engine

    def engine_size(self):
        return self.engine.size


class EconomicalEngine:
    size = 1.0


class SportEngine:
    size = 3.5


car = Car(SportEngine())
print(car.engine_size())

In this example our Car has an engine but we do not define an Engine type, instead we simply state that the Car expects a type with a size variable. This does ofcourse means we could pass in anything with a size variable (which ofcourse might not always make much sense).

Static/class methods

Up until now we’ve seen instance methods on classes. Python also includes two decorators that can be used in conjunction with our methods to make them acts in a similar way to static methods.

The first of these is the decorator @staticmethod, for example

class Dog:
    @staticmethod
    def animal_type():
        return "Dog";

# and we call the method like this
print(Dog.animal_type())
# or
d = Dog()
print(d.animal_type())

Python also includes another static-like method type which is decorated with @classmethod. A class method differs from a static method in that it includes an implicit cls variable (by convention cls is used for the first argument). For example

class Dog:
    @classmethod
    def name(cls):
        return "English Springer Spaniel"

# and we call the method like this
print(Dog.name())
# or
d = Dog()
print(d.name())

So what’s the difference? Both appear to allow us to work in a “static” method way but the @classmethod allows us to still get an instance of the class, but it appears that this is almost like passing an instance of a new class into the method, i.e. these appear to be equivalent. Let’s assume we have this

class Dog:
    _name = "Dog"

    @classmethod
    def name(cls):
        print(cls)
        return cls._name

d = Dog()
d._name = "English Springer Spaniel"

print(d.name)
# same as
print(Dog().name)

In the above we change the instance variable _name but both print statements will output “Dog” so appear functionally equivalent.