{"id":5634,"date":"2021-06-02T10:06:18","date_gmt":"2021-06-02T08:06:18","guid":{"rendered":"https:\/\/joapen.com\/blog\/?p=5634"},"modified":"2021-06-02T10:15:30","modified_gmt":"2021-06-02T08:15:30","slug":"data-analysis-on-market-breadth-data","status":"publish","type":"post","link":"http:\/\/joapen.com\/blog\/2021\/06\/02\/data-analysis-on-market-breadth-data\/","title":{"rendered":"Data analysis on market breadth data"},"content":{"rendered":"\n

This post is an exercise to learn how to predict using different data on a machine learning model.<\/p>\n\n\n\n

Background<\/h2>\n\n\n\n

Market breadth<\/a> data and indicators are very popular in the investment world. I find them useful, and as I know them, I will use them as basis to experiment machine learning models.<\/p>\n\n\n\n

Data Analysis<\/h2>\n\n\n\n

I have done the clean up of data in excel and then I have moved to a .CSV. There I have done a basic visual analysis on a Jupyter notebook. The result of the analysis can be found on this GitHub repository<\/a>.<\/p>\n\n\n\n

The correlation matrix of the raw data is:<\/h3>\n\n\n\n
\"\"\/<\/figure>\n\n\n\n

I have created a relative matrix with the data, but the basic correlation is worse than the original one.<\/p>\n\n\n\n

Applying models on the data<\/h2>\n\n\n\n

I have used basic regressor from XGBoost, and I have created 10 basic models changing the parameters a little bit, looking for a better mean absolute error.<\/p>\n\n\n\n

from xgboost import XGBRegressor\r\n\r\nmodel_1 = XGBRegressor(n_estimators=100, learning_rate=0.05)\r\nmodel_2 = XGBRegressor(n_estimators=500, learning_rate=0.05)\r\nmodel_3 = XGBRegressor(n_estimators=1000, learning_rate=0.05)\r\nmodel_4 = XGBRegressor(n_estimators=2000, learning_rate=0.05)\r\nmodel_5 = XGBRegressor(n_estimators=3000, learning_rate=0.05)\r\nmodel_6 = XGBRegressor(n_estimators=100, learning_rate=0.1)\r\nmodel_7 = XGBRegressor(n_estimators=500, learning_rate=0.1)\r\nmodel_8 = XGBRegressor(n_estimators=1000, learning_rate=0.1)\r\nmodel_9 = XGBRegressor(n_estimators=2000, learning_rate=0.1)\r\nmodel_10 = XGBRegressor(n_estimators=3000, learning_rate=0.1)\r\n\r\nmodels = [model_1, model_2, model_3, model_4, model_5, model_6, model_7, model_8, model_9, model_10]<\/code><\/pre>\n\n\n\n
Model 3 was the one with lower MAE, so I picked this one. Then I did the prediction and I plotted it. This was the result of the prediction on test data (from 1\/1\/2020 to 25\/4\/2021):<\/p>\n\n\n\n
\"\"\/
Result of model 3 prediction using a XGBoost regressor<\/figcaption><\/figure>\n\n\n\n
The result is very funny, the model is able to predict the part of 2020 that was horrible to the investors (February – May) but the rest of the time is incapable to hit anything.<\/p>\n\n\n\n
You can see the complete code and charts here:<\/p>\n\n\n\n
https:\/\/github.com\/joapen\/ML-Learning-bucket\/blob\/main\/marketbreadthdata_v2.ipynb<\/a><\/p>\n\n\n\n
I will continue working on this, this is being funny.<\/p>\n","protected":false},"excerpt":{"rendered":"
This post is an exercise to learn how to predict using different data on a machine learning model. Background Market breadth data and indicators are very popular in the investment world. I find them useful, and as I know them, I will use them as basis to experiment machine learning models. Data Analysis I have … Read more<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[151,104],"tags":[],"class_list":["post-5634","post","type-post","status-publish","format-standard","hentry","category-machine-learning","category-python"],"yoast_head":"\nData analysis on market breadth data -<\/title>\n<meta name=\"description\" content=\"This post is an exercise to learn how to predict using different data on a machine learning model. Background Market breadth data and indicators are very - joapen projects\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"http:\/\/joapen.com\/blog\/2021\/06\/02\/data-analysis-on-market-breadth-data\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Data analysis on market breadth data -\" \/>\n<meta property=\"og:description\" content=\"This post is an exercise to learn how to predict using different data on a machine learning model. Background Market breadth data and indicators are very - joapen projects\" \/>\n<meta property=\"og:url\" content=\"http:\/\/joapen.com\/blog\/2021\/06\/02\/data-analysis-on-market-breadth-data\/\" \/>\n<meta property=\"og:site_name\" content=\"joapen projects\" \/>\n<meta property=\"article:published_time\" content=\"2021-06-02T08:06:18+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2021-06-02T08:15:30+00:00\" \/>\n<meta name=\"author\" content=\"joapen\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"joapen\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"2 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"http:\\\/\\\/joapen.com\\\/blog\\\/2021\\\/06\\\/02\\\/data-analysis-on-market-breadth-data\\\/#article\",\"isPartOf\":{\"@id\":\"http:\\\/\\\/joapen.com\\\/blog\\\/2021\\\/06\\\/02\\\/data-analysis-on-market-breadth-data\\\/\"},\"author\":{\"name\":\"joapen\",\"@id\":\"https:\\\/\\\/joapen.com\\\/blog\\\/#\\\/schema\\\/person\\\/23919df2312175fe9c4609203595b217\"},\"headline\":\"Data analysis on market breadth data\",\"datePublished\":\"2021-06-02T08:06:18+00:00\",\"dateModified\":\"2021-06-02T08:15:30+00:00\",\"mainEntityOfPage\":{\"@id\":\"http:\\\/\\\/joapen.com\\\/blog\\\/2021\\\/06\\\/02\\\/data-analysis-on-market-breadth-data\\\/\"},\"wordCount\":263,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\\\/\\\/joapen.com\\\/blog\\\/#\\\/schema\\\/person\\\/23919df2312175fe9c4609203595b217\"},\"articleSection\":[\"Machine Learning\",\"Python\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"http:\\\/\\\/joapen.com\\\/blog\\\/2021\\\/06\\\/02\\\/data-analysis-on-market-breadth-data\\\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"http:\\\/\\\/joapen.com\\\/blog\\\/2021\\\/06\\\/02\\\/data-analysis-on-market-breadth-data\\\/\",\"url\":\"http:\\\/\\\/joapen.com\\\/blog\\\/2021\\\/06\\\/02\\\/data-analysis-on-market-breadth-data\\\/\",\"name\":\"Data analysis on market breadth data -\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/joapen.com\\\/blog\\\/#website\"},\"datePublished\":\"2021-06-02T08:06:18+00:00\",\"dateModified\":\"2021-06-02T08:15:30+00:00\",\"description\":\"This post is an exercise to learn how to predict using different data on a machine learning model. Background Market breadth data and indicators are very - joapen projects\",\"breadcrumb\":{\"@id\":\"http:\\\/\\\/joapen.com\\\/blog\\\/2021\\\/06\\\/02\\\/data-analysis-on-market-breadth-data\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"http:\\\/\\\/joapen.com\\\/blog\\\/2021\\\/06\\\/02\\\/data-analysis-on-market-breadth-data\\\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"http:\\\/\\\/joapen.com\\\/blog\\\/2021\\\/06\\\/02\\\/data-analysis-on-market-breadth-data\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/joapen.com\\\/blog\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Data analysis on market breadth data\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/joapen.com\\\/blog\\\/#website\",\"url\":\"https:\\\/\\\/joapen.com\\\/blog\\\/\",\"name\":\"joapen projects\",\"description\":\"Just a place to write\",\"publisher\":{\"@id\":\"https:\\\/\\\/joapen.com\\\/blog\\\/#\\\/schema\\\/person\\\/23919df2312175fe9c4609203595b217\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/joapen.com\\\/blog\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":[\"Person\",\"Organization\"],\"@id\":\"https:\\\/\\\/joapen.com\\\/blog\\\/#\\\/schema\\\/person\\\/23919df2312175fe9c4609203595b217\",\"name\":\"joapen\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/joapen.com\\\/blog\\\/wp-content\\\/uploads\\\/2021\\\/04\\\/joapen-mini.jpeg\",\"url\":\"https:\\\/\\\/joapen.com\\\/blog\\\/wp-content\\\/uploads\\\/2021\\\/04\\\/joapen-mini.jpeg\",\"contentUrl\":\"https:\\\/\\\/joapen.com\\\/blog\\\/wp-content\\\/uploads\\\/2021\\\/04\\\/joapen-mini.jpeg\",\"width\":400,\"height\":400,\"caption\":\"joapen\"},\"logo\":{\"@id\":\"https:\\\/\\\/joapen.com\\\/blog\\\/wp-content\\\/uploads\\\/2021\\\/04\\\/joapen-mini.jpeg\"},\"sameAs\":[\"http:\\\/\\\/www.joapen.com\"]}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Data analysis on market breadth data -","description":"This post is an exercise to learn how to predict using different data on a machine learning model. Background Market breadth data and indicators are very - joapen projects","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"http:\/\/joapen.com\/blog\/2021\/06\/02\/data-analysis-on-market-breadth-data\/","og_locale":"en_US","og_type":"article","og_title":"Data analysis on market breadth data -","og_description":"This post is an exercise to learn how to predict using different data on a machine learning model. Background Market breadth data and indicators are very - joapen projects","og_url":"http:\/\/joapen.com\/blog\/2021\/06\/02\/data-analysis-on-market-breadth-data\/","og_site_name":"joapen projects","article_published_time":"2021-06-02T08:06:18+00:00","article_modified_time":"2021-06-02T08:15:30+00:00","author":"joapen","twitter_misc":{"Written by":"joapen","Est. reading time":"2 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"http:\/\/joapen.com\/blog\/2021\/06\/02\/data-analysis-on-market-breadth-data\/#article","isPartOf":{"@id":"http:\/\/joapen.com\/blog\/2021\/06\/02\/data-analysis-on-market-breadth-data\/"},"author":{"name":"joapen","@id":"https:\/\/joapen.com\/blog\/#\/schema\/person\/23919df2312175fe9c4609203595b217"},"headline":"Data analysis on market breadth data","datePublished":"2021-06-02T08:06:18+00:00","dateModified":"2021-06-02T08:15:30+00:00","mainEntityOfPage":{"@id":"http:\/\/joapen.com\/blog\/2021\/06\/02\/data-analysis-on-market-breadth-data\/"},"wordCount":263,"commentCount":0,"publisher":{"@id":"https:\/\/joapen.com\/blog\/#\/schema\/person\/23919df2312175fe9c4609203595b217"},"articleSection":["Machine Learning","Python"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["http:\/\/joapen.com\/blog\/2021\/06\/02\/data-analysis-on-market-breadth-data\/#respond"]}]},{"@type":"WebPage","@id":"http:\/\/joapen.com\/blog\/2021\/06\/02\/data-analysis-on-market-breadth-data\/","url":"http:\/\/joapen.com\/blog\/2021\/06\/02\/data-analysis-on-market-breadth-data\/","name":"Data analysis on market breadth data -","isPartOf":{"@id":"https:\/\/joapen.com\/blog\/#website"},"datePublished":"2021-06-02T08:06:18+00:00","dateModified":"2021-06-02T08:15:30+00:00","description":"This post is an exercise to learn how to predict using different data on a machine learning model. Background Market breadth data and indicators are very - joapen projects","breadcrumb":{"@id":"http:\/\/joapen.com\/blog\/2021\/06\/02\/data-analysis-on-market-breadth-data\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["http:\/\/joapen.com\/blog\/2021\/06\/02\/data-analysis-on-market-breadth-data\/"]}]},{"@type":"BreadcrumbList","@id":"http:\/\/joapen.com\/blog\/2021\/06\/02\/data-analysis-on-market-breadth-data\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/joapen.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Data analysis on market breadth data"}]},{"@type":"WebSite","@id":"https:\/\/joapen.com\/blog\/#website","url":"https:\/\/joapen.com\/blog\/","name":"joapen projects","description":"Just a place to write","publisher":{"@id":"https:\/\/joapen.com\/blog\/#\/schema\/person\/23919df2312175fe9c4609203595b217"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/joapen.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":["Person","Organization"],"@id":"https:\/\/joapen.com\/blog\/#\/schema\/person\/23919df2312175fe9c4609203595b217","name":"joapen","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/joapen.com\/blog\/wp-content\/uploads\/2021\/04\/joapen-mini.jpeg","url":"https:\/\/joapen.com\/blog\/wp-content\/uploads\/2021\/04\/joapen-mini.jpeg","contentUrl":"https:\/\/joapen.com\/blog\/wp-content\/uploads\/2021\/04\/joapen-mini.jpeg","width":400,"height":400,"caption":"joapen"},"logo":{"@id":"https:\/\/joapen.com\/blog\/wp-content\/uploads\/2021\/04\/joapen-mini.jpeg"},"sameAs":["http:\/\/www.joapen.com"]}]}},"_links":{"self":[{"href":"http:\/\/joapen.com\/blog\/wp-json\/wp\/v2\/posts\/5634","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/joapen.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/joapen.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/joapen.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/joapen.com\/blog\/wp-json\/wp\/v2\/comments?post=5634"}],"version-history":[{"count":2,"href":"http:\/\/joapen.com\/blog\/wp-json\/wp\/v2\/posts\/5634\/revisions"}],"predecessor-version":[{"id":5637,"href":"http:\/\/joapen.com\/blog\/wp-json\/wp\/v2\/posts\/5634\/revisions\/5637"}],"wp:attachment":[{"href":"http:\/\/joapen.com\/blog\/wp-json\/wp\/v2\/media?parent=5634"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/joapen.com\/blog\/wp-json\/wp\/v2\/categories?post=5634"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/joapen.com\/blog\/wp-json\/wp\/v2\/tags?post=5634"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}