How to import data with genfromtxt?

Numpy provides several functions to create arrays from tabular data.
genfromtxt runs two main loops:
1.The first loop converts each line of the file in a sequence of strings.
2.The second loop converts each string to the appropriate data type.

Syntax:
numpy.genfromtxt(fname, dtype=<type 'float'>, comments='#', delimiter=None, skip_header=0, skip_footer=0, converters=None, missing_values=None, filling_values=None, usecols=None, names=None, excludelist=None, deletechars=None, replace_space='_', autostrip=False, case_sensitive=True, defaultfmt='f%i', unpack=None, usemask=False, loose=True, invalid_raise=True, max_rows=None)

This mechanism is slower than a single loop, but it gives more flexibility. In particular, "genfromtxt" is able to take missing data into account, when other faster and simpler functions like "loadtxt" cannot.

Process:
1.Defining the Input:
The input file can be a text file or an archive

2.Splitting the lines into columns:
The delimiter argument
Example:
>>> data = "1, 2, 3\n4, 5, 6"
>>> np.genfromtxt(StringIO(data), delimiter=',')
array([[ 1.,  2.,  3.],
       [ 4.,  5.,  6.]])


>>> data = "  1  2  3\n  4  5 67\n890123  4"
>>> np.genfromtxt(StringIO(data), delimiter=3)
array([[   1.,    2.,    3.],
       [   4.,    5.,   67.],
       [ 890.,  123.,    4.]])


>>> data = "123456789\n   4  7 9\n   4567 9"
>>> np.genfromtxt(StringIO(data), delimiter=(4, 3, 2))
array([[ 1234.,   567.,    89.],
       [    4.,     7.,     9.],
       [    4.,   567.,     9.]])

The autostrip argument

>>> data = "1, abc , 2\n 3, xxx, 4"

>>> np.genfromtxt(StringIO(data), delimiter=",", dtype="|S5")

array([['1', ' abc ', ' 2'],

       ['3', ' xxx', ' 4']], 

      dtype='|S5')

>>> np.genfromtxt(StringIO(data), delimiter=",", dtype="|S5", autostrip=True)

array([['1', 'abc', '2'],

       ['3', 'xxx', '4']], 

      dtype='|S5')

The comments argument

>> data = """#

... # Skip me!

... #Skip me too!

... 1,2

... 3,4

... 5,6

... 7,8

... #And here comes the last line

... 9,0

... """

>>> np.genfromtxt(StringIO(data), comments="#", delimiter=",")

array([[ 1.,  2.],

       [ 3.,  4.],

       [ 5.,  6.],

       [ 7.,  8.],

       [ 9.,  0.]])

Skipping lines and choosing columns

The skip_header and skip_footer arguments

The presence of a header in the file can hinder data processing. In that case, we need to use the skip_header optional argument. The values of this argument must be an integer which corresponds to the number of lines to skip at the beginning of the file, before any other action is performed. Similarly, we can skip the last n lines of the file by using the skip_footer attribute and giving it a value of n:

 data = "\n".join(str(i) for i in range(10))

>>> np.genfromtxt(StringIO(data),)

array([ 0.,  1.,  2.,  3.,  4.,  5.,  6.,  7.,  8.,  9.])

>>> np.genfromtxt(StringIO(data),

...              skip_header=3, skip_footer=5)

array([ 3.,  4.])

The usecols argument:

In some cases, we are not interested in all the columns of the data but only a few of them. We can select which columns to import with the usecols argument. This argument accepts a single integer or a sequence of integers corresponding to the indices of the columns to import. Remember that by convention, the first column has an index of 0. Negative integers behave the same as regular Python negative indexes.

>>> data = "1 2 3\n4 5 6"

>>> data

'1 2 3\n4 5 6'

>>> np.genfromtxt(StringIO(data), usecols=(0, -1))

array([[ 1.,  3.],

       [ 4.,  6.]])

If the columns have names, we can also select which columns to import by giving their name to the usecols argument, either as a sequence of strings or a comma-separated string:

>>> data = "1 2 3\n4 5 6"

>>> np.genfromtxt(StringIO(data),names="a, b, c", usecols=("a", "c"))

array([(1.0, 3.0), (4.0, 6.0)], 

      dtype=[('a', '<f8'), ('c', '<f8')])

Setting the names

The names argument

>>> data = StringIO("1 2 3\n 4 5 6")

>>> data 

<StringIO.StringIO instance at 0x7f38ac086680>

>>> np.genfromtxt(data, dtype=[(_, int) for _ in "abc"])

array([(1, 2, 3), (4, 5, 6)], 

      dtype=[('a', '<i8'), ('b', '<i8'), ('c', '<i8')])

>>> data = StringIO("1 2 3\n 4 5 6")

>>> np.genfromtxt(data, names="A, B, C")

array([(1.0, 2.0, 3.0), (4.0, 5.0, 6.0)], 

      dtype=[('A', '<f8'), ('B', '<f8'), ('C', '<f8')])

>>> data = StringIO("So it goes\n#a b c\n1 2 3\n 4 5 6")

>>> np.genfromtxt(data, skip_header=1, names=True)

array([(1.0, 2.0, 3.0), (4.0, 5.0, 6.0)], 

      dtype=[('a', '<f8'), ('b', '<f8'), ('c', '<f8')])

The defaultfmt argument

If names=None but a structured dtype is expected, names are defined with the standard NumPy default of "f%i", yielding names like f0, f1 and so forth:

>>> data = StringIO("1 2 3\n 4 5 6")

>>> np.genfromtxt(data, dtype=(int, float, int))

array([(1, 2.0, 3), (4, 5.0, 6)], 

      dtype=[('f0', '<i8'), ('f1', '<f8'), ('f2', '<i8')])

>>> data = StringIO("1 2 3\n 4 5 6")

>> data

<StringIO.StringIO instance at 0x7f38a709fc20>

>>> np.genfromtxt(data, dtype=(int, float, int), names="a")

array([(1, 2.0, 3), (4, 5.0, 6)], 

      dtype=[('a', '<i8'), ('f0', '<f8'), ('f1', '<i8')])

How do we create a Numpy array?

Numpy has built-in functions for creating arrays.

zeros(shape) will create an array filled with "0" values with the specified shape. The default dtype is float64.
Example:
>>> np.zeros((2, 3))
array([[ 0.,  0.,  0.],

       [ 0.,  0.,  0.]])

ones(shape) will create an array filled with 1 values.

Example:

>>> np.ones((2,3))

array([[ 1.,  1.,  1.],

       [ 1.,  1.,  1.]])

arange() will create arrays with regular increment values

Example:

>>> np.arange(10)

array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9])

>>> np.arange(2, 10, dtype=np.float)

array([ 2.,  3.,  4.,  5.,  6.,  7.,  8.,  9.])

>>> np.arange(2, 3, 0.1)

array([ 2. ,  2.1,  2.2,  2.3,  2.4,  2.5,  2.6,  2.7,  2.8,  2.9])

linspace() will create arrays with a specified number of elements, and spaced equally between the specified beginning and end values.

Example:

>>> np.linspace(1., 4., 6)

array([ 1. ,  1.6,  2.2,  2.8,  3.4,  4. ])

indices() will create a set of arrays (stacked as a one-higher dimensioned array), one per dimension with each representing variation in that dimension.

Example:

>>> np.indices((3,3))

array([[[0, 0, 0],

        [1, 1, 1],

        [2, 2, 2]],

       [[0, 1, 2],

        [0, 1, 2],

        [0, 1, 2]]])

How to convert Python array_like Objects to Numpy Arrays?

In general, numerical data arranged in an array-like structure(example: lists and tuples.) in Python can be converted to arrays by the use of the array() function.

Example:
>>> x = np.array([2,3,1,0])
>>> x
array([2, 3, 1, 0])

>>> x = np.array([[1,2.0],[0,0],(1+1j,3.)])

>>> x

array([[ 1.+0.j,  2.+0.j],

       [ 0.+0.j,  0.+0.j],

       [ 1.+1.j,  3.+0.j]])

>>> x = np.array([[ 1.+0.j, 2.+0.j], [ 0.+0.j, 0.+0.j], [ 1.+1.j, 3.+0.j]])

>>> x

array([[ 1.+0.j,  2.+0.j],

       [ 0.+0.j,  0.+0.j],

       [ 1.+1.j,  3.+0.j]])

Sort a python Dictionary by Value

You cannot  sort a dictionary, but only get a representation of a dictionary that is sorted. Dictionary are inherently order-less, but other types such as lists and tuples, are not. So you need a sorted representation, which will be a list—probably a list of tuples.

Sort a python Dictionary by value:

import operator
x = {1: 2, 3: 4, 4: 3, 2: 1, 0: 0}
sorted_x = sorted(x.items(), key=operator.itemgetter(1))

Sort a python Dictionary by Key:
import operator
x = {1: 2, 3: 4, 4: 3, 2: 1, 0: 0}
sorted_x = sorted(x.items(), key=operator.itemgetter(0))

What is Bubble sort?

 It compares adjacent items and exchanges those that are out of order.

Example:
x=[47, 98, 85, 85, 67, 67]
def bubble_sort(x):
for i in range(len(x)-1,0,-1):
for j in range(i):
if x[j]>x[j+1]:
temp=x[j]
x[j]=x[j+1]
x[j+1]=temp
return x

>>> print(bubble_sort(x))
[47, 67, 67, 85, 85, 98]

Insertion Sort in Python

Insertion sort is a simple sorting algorithm that builds the final sorted  list one item at a time. It is much less efficient.

Best-case performance: O(n) comparisons,
Average performance:О(n2) comparisons
Worst-case space complexity:О(n) total

Example:
x=[98,67,85,67,85,47]
>>> def insertion_sort(x):
for i in range(1,len(x)):
j=i
while j>0 and x[j]<x[j-1]:
x[j],x[j-1]=x[j-1],x[j]
j=j-1
return k

>>> insertion_sort(x)

[6, 7, 12, 32, 43, 45]

Quicksort in Python

Quicksort is divide and conquer sorting algorithm.

Working:
A quick sort first selects a value, which is called the pivot value.
The role of the pivot value is to assist with splitting the list.
The actual position where the pivot value belongs in the final sorted list, commonly called the split point, will be used to divide the list for subsequent calls to the quick sort.
Expected Running time:O(nlogn)
Worstcase :O(n^2)

Advantages:
1.Good Cache Performance
2.Good Parallel Performance


Quick sort has two phase:
1.Partition Phase
def partition(myList, start, end):
    pivot = myList[start]
    left = start+1
    right = end
    done = False
    while not done:
        while left <= right and myList[left] <= pivot:
            left = left + 1
        while myList[right] >= pivot and right >=left:
            right = right -1
        if right < left:
            done= True
        else:
            # swap places
            temp=myList[left]
            myList[left]=myList[right]
            myList[right]=temp
    # swap start with myList[right]
    temp=myList[start]
    myList[start]=myList[right]
    myList[right]=temp
    return right

2.Sort Phase
def quicksort(myList, start, end):
    if start < end:
        # partition the list
        pivot = partition(myList, start, end)
        # sort both halves
        quicksort(myList, start, pivot-1)
        quicksort(myList, pivot+1, end)
    return myList