Open-source Twitter Intelligence

I continue being curious about OSINT, and I found this project named Open-source Twitter Intelligence in GIT Hub .

It’s a set of code that do scraping on Twitter without using Twitter’s API, allowing you to scrape a user’s followers, following, Tweets and more.

Apparently Twitter’s API has some limits and this approach avoid them.

It’s written in Perl, so I cloned with PyCharm, I uploaded some libraries and I created a test.py file that enabled me to test it:

import twint

c = twint.Config()

c.Username = "joa_pen"
c.Custom["tweet"] = ["id"]
c.Custom["user"] = ["bio"]
c.Limit = 10
c.Store_csv = True
c.Output = "none"

twint.run.Search(c)

 

 

Open-source intelligence, first try

Leyendo documentación sobre seguridad y tratando de entender como se ponen en práctica los conceptos de OSINT, encontré un vídeo sencillo de como usar EO-Ripper.py para rastrear correos electrónicos y sus contraseñas:

Me pareció curioso probarlo, de manera que encontré un hueco días más tarde.

Los pasos que di esta vez han sido:

  • Instalar python 3.6.
  • He instalado PyCharm ( https://www.jetbrains.com/) en vez de Eclipse, así pruebo un entorno distinto.
  • Me pide instalar las siguientes librerías:
    • beautifulsoup4, que viene en PyCharm
    • mechanize: solo disponible para python 2.x
    • cssselect: que viene en PyCharm
    • cookielib, no la encuentro
  • Instalo GIT (https://git-scm.com/download/gui/windows) para poder clonar el proyecto.
  • Instalé el proyecto https://github.com/thibauts/duckduckgo,
  • He creado un fichero de test para el duckduckgo y funciona.
  • No se sido capaz de encontrar alternativas rápidas al hecho que el código está preparado para python 2.7 y he instalado python 3.6, con lo que he desistido.
  • Necesito encontrar más tiempo, lo aparco aquí.

 

Quantitative trading on cryptocurrency market Q3

This is the second chapter of a learning process that started last September.

Third Quarter

The third step is defined for the next 3 months, where the main goal is to define a specific strategy of quantitative trading and work on it with real money on crypto currency market.

Following the V2MOM model:

  • Vision: Have a strategy running in crypto currency market running not with a period of 2 – 3 hours, but some days (stop operating at 3m).
  • Values: have fun, learn a lot, build a team with Dani, do practices and more practices.
  • Method: learn about trading basis, do backtesting with Quantopian on stocks or Forex (analyze the results in deep).
  • Obstacles: Time.
  • Measures:
    • Make short/long decisions based on 1 hour.
    • Read at least 1 book of trading.
    • Perform backtesting with Quantopian and document the results and findings.
    • Improve and document the “mode operations” and “mode backtesting”.

Death line = June 2018

Results (July 1st, 2018)

  • Time to be accountable, let’s go…
  • I have done more than 50 operations, being May and June with negative results. July has been with 34 operations and positive global results.
  • I have learned to trade with a bearish market, and interestingly I have issues to work with a bullish market (I short too soon).
  • I have not worked with Quantopian nor tradingview on any backtesting, this was a bid fault.
  • I discovered an interesting indicator: VPCI, Volume price confirmation indicator. It really helps to indentify real moves of the market.
  • I have been able to cultivate patience during these months, but some days I did some moves that did not make sense. I have to evolve on this.
  • I finish read one of the fundamental analysis books.
  • I have started to apply the knowledge I’m acquiring on the medium term market I work on stocks.
  • Not too much.
  • I need to retake the back testing exercises and continuing the practices.

Guru99, python and more

There are so many tutorials about almost everything, this time I found this one: https://www.guru99.com

  • Guru99 Vision: Fun & Free Education for ALL
  • Guru99 Mission: To bring all feasible courses , online.

Python was the one who took the first look, but I went quickly to other tutorials.

I am inmersed now on a SAP HCM project, so I have read the basis of the SAP Infotype organization and other relevant stuff that the consultants are mentioning every single day.

Read tutorials is hard, but these ones are in real “plain english” so you quickly understand the topics. They are focused on people without knowledge of a topic.

Anaconda, configuration

Python Data Science Platform

I have installed this Anaconda environment with the purpose of performing the backtesting activities I’m working on with Python.

The process

It’s easy to install and there is nothing you have to configure. You just need to start coding and working on it.

The user adoption for the notebook is quick, I just had to watch a couple of videos, it’s very straightforward.

The unexpected outcome (March / 2018)

After some days practicing with Notebook I started learning about Quantopian basis. Quantopian toolset is composed by an IDE and an IPython notebook, which is equal to Jupyter notebook.

Quantitative trading on cryptocurrency market Q2

This is the second chapter of a learning process that started last September.

Second Quarter

The second step is defined for the next 3 months, where the main goal is to define a specific strategy of quantitative trading and work on it with real money on crypto currency market.

Following the V2MOM model:

  • Vision: Have a strategy running in crypto currency market.
  • Values: have fun, learn a lot, build a team with Dani, do practices and more practices.
  • Method: learn about quantitative trading principles and patterns, understand the basis of how blockchain markets work.
  • Obstacles: Time.
  • Measures: have an account with money and making short/long decisions. Have a lot of data related to the backtesting of different versions of a specific strategy. Accumulate a list of learned lessons and experiences for blockchain, quantitative trading and python.

Death line = March 2018

Results (April 1st, 2018)

  • Time to be accountable, let’s go…
  • I have traded more than 50 times on LTCEUR and ETHEUR. It’s very
  • When you lose, you learn something; when you win, you do not learn anything.
  • We have added a thread of lessons learned in the real life of the trader with insights, habits, and other tips.
  • We have learned about the need to keep perspective, read the news about the coins and respect the basic rules.
  • We have failed in the basic habits and we have learned which triggers where no happening when they should happen (the more important one: set stop-loss).
  • I’m still running long-shorts in minutes (3 minutes – 7 minutes), I need to start opening positions at hours rate and be able to put in action an strategy within 20 to 40 hours with an opened position. This is in some way as to play chess: quick games and long games.
  • We have learned how to set time-framing exercise before to start trading, draw supports and resistance lines, look for volumes, and other basic things. We need to organize them and value them properly, but they are in our mind already.
  • I have read a book, Machine trading from Ernie Chang, that has enabled me to understand the next steps that we should take during the next quarters.
  • I have learned a lot of basic concepts on Machine Learning , the basis. If tomorrow someone tell me to manage a project related to that, I know the basis and I will offer myself to manage it.
  • I have learned about Quantopian, basic concepts, how it works, how powerfull the environment is, and I have done a couple of backtests. There is a lot of information to read, basic statistics I still need to learn and the most important thing: I need to evolve a scientific mindset.
  • I have learned about bias, doing operating on the real environment and reading “Thinking Basketball“; to see the things in a different context provides a valuable insight.

For next quarter I have some ideas, but I still have not been able to concrete what should be the right focus to cover.

Hello world in quantitative trading

Since I started on this field the focus has been to be able to build a set of

I found this training course with the slides posted, where Michael Halls-Moore explains the basis of the backtesting with Python and Pandas.

Things to remind

  • Taxonomy of trading strategies: forecasting, mean reversion, momentum and high frequency trading.
  • Review of the backtesting pitfalls: the same ones that mentioned by Ernie Chang.
  • Things to do after you perform the moving average crossover strategy:
    • Multi-symbol portfolio.
    • Risk management (the most important topic).
    • True event-driven backtesting, realistic handling of transaction costs, fees…

Moving Average Crossover Strategy

this is defined as the “Hello World” of quantitative trading. It’s included a specific article,

Exponentially weighted moving average (EWMA)

EWMA is a type of infinite impulse response filter that applies weighting factors which decrease exponentially. The weighting for each older datum decreases exponentially, never reaching zero.

Code written in Python 2

The proposed code is in python 2, but here you can find a guy that adapted it to python3. It uses Quandl library for obtaining financial data easily.

 

 

Python, Good morning

Good morning is the name of a library that enables you to download data from stocks related to fundamentals.

I have uploaded the GIT file on my eclipse and I have shown some basic data in a data frame. This was something initially easy.

My goal is to draw data in a monthly basis for these fundamentals and be able to compare with other datasets. By the moment I’m not able to show data by at least in a monthly basis.

The code of my basic test:

import good_morning as gm

kr = gm.KeyRatiosDownloader()
frames = kr.download(‘T’)

count = 0
frame_size = len(frames)
while (count < frame_size):
         print( ‘The count is:’, count)
print(frames[count])

count = count + 1

print(“Good bye!”)

Quantitative trading strategy: Pandas and matplotlib

Start from basis is important to me to understand how to handle basic data and start having real contact with the data and the code.

I found this tutorial very useful for these purposes. The use of Pandas for reading data from yahoo, google… and matplotlib to build an easy chart is key to take the first steps. The best of the tutorial are the comments about each step.

Problems I have found

Yahoo has closed the API that enabled Pandas to retrieve data. My colleague has found a workaround to continue using the API. The solution consists on:

  1. Add with pip the library: fix_yahoo_finance.
  2. Add these 2 lines to the code:

import fix_yahoo_finance as yf

yf.pdr_override() # <== that’s all it takes 🙂

This video is a must see to me as Jev Kuznetsov explains it from scratch.

The first code I performed was this one:

from pandas_datareader import data
import pandas as pd
import matplotlib.pyplot as plt

# Define the instruments to download. We would like to see Apple, Microsoft and the S&P500 index.
tickers = [‘T’, ‘VZ’, ‘SPY’]

# Define which online source one should use
data_source = ‘google’

# We would like all available data from 01/01/2000 until 12/31/2016.
start_date = ‘2015-01-01’
end_date = ‘2017-10-10’

# User pandas_reader.data.DataReader to load the desired data. As simple as that.
panel_data = data.DataReader(tickers, data_source, start_date, end_date)

# Getting just the adjusted closing prices. This will return a Pandas DataFrame
# The index in this DataFrame is the major index of the panel_data.
close = panel_data.ix[‘Close’]

# Getting all weekdays between 01/01/2000 and 12/31/2016
all_weekdays = pd.date_range(start=start_date, end=end_date, freq=’B’)

# How do we align the existing prices in adj_close with our new set of dates?
# All we need to do is reindex close using all_weekdays as the new index
close = close.reindex(all_weekdays)
#print(close.head(-100))
print(close.describe())
close = close.fillna(method=’ffill’)
# Get the MSFT time series. This now returns a Pandas Series object indexed by date.
vz = close.ix[:, ‘VZ’]
spy = close.ix[:, ‘SPY’]
# Calculate the 20 and 100 days moving averages of the closing prices
short_rolling_vz = vz.rolling(window=20).mean()
long_rolling_vz = vz.rolling(window=100).mean()

# Plot everything by leveraging the very powerful matplotlib package
fig = plt.figure()
ax = fig.add_subplot(1,1,1)
ax.plot(vz.index, vz, label=’VZ’)
#ax.plot(spy.index, spy, label=’SPY’)
ax.plot(short_rolling_vz.index, short_rolling_vz, label=’20 days rolling’)
ax.plot(long_rolling_vz.index, long_rolling_vz, label=’100 days rolling’)
ax.set_xlabel(‘Date’)
ax.set_ylabel(‘Closing price ($)’)
ax.legend()
plt.show()

Pydev with Eclipse

I find EMACS too much complex for me. I want to concentrate on the analysis of data with Pyhton, so I have looked for and alternative IDE for Python: Pydev.

The steps I have followed are:

  1. Download and install Eclipse (in reality I had it on my computer).
  2. Then, download, install and test Phyton.
  3. Once done, download and install Jython.
  4. Download and install IronPython.
  5. Install PyDev.
  6. Create a new PyDev project.
  7. I have created a hello.py file with print(“hello world”), then I have debugged it and it worked.
  8. Create a package to gather all the information.
  9. The set-up of the environment is very well explained. You first have to configure the interpreter, then you can develop all you need. In my case I’m starting importing some finance data from Yahoo Finance.
  10. Using Windows>Properties>Python Interpreter , install library “numpy” using “pip” as installer. This step is crucial to me as Python interpreter just work on PyDev in my computer.

EMACS

I’m sure EMACS runs faster and it’s more efficient than Eclipse, but the point is that now I can concentrate on the development of the scripts I want to implement. 🙂