texts: Hypertune xgboost to dealing with imbalanced dataset

texts:
My training data has extremely class imbalanced {0:872525,1:3335} with 100 features. I use xgboost to build classification model with bayessian optimisation to hypertune the model in range
{learning rate:(0.001,0.1), min_split_loss:(0.10), max_depth:(3,70), min_child_weight:(1:20), max_delta_step:(1,20), subsample:(0:1), colsample_bytree:(0.5,1), lambda:(0,10), alpha:(0,10), scale_pos_weight:(1,262), n_estimator:(1,20)}.
I also use binary:logistics as the objective model and roc_auc as the metrics with booster gbtree.
The cross validation score is 82.5%.
However, when I implemented the model to the testing data I got the score only
Roc_auc: 75.2%, pr_auc: 15%, log_loss: 0.046, and confusion matrix: [[19300 7],[103 14]].
I need helping to find the best way to increase the true possitive with tolerance false positive until 3 times actual positive.


texts:
My data set contains the pickup latitude, pickup longitude of the customer and drivers current location latitude and longitude.
My aim is to distribute a set of  M or M-x (in the case where shortage of drivers)
for N customers. The driver has to be chosen with less distance and less ETA.

Any solution in data science with python or R?

customer_id  pickuplocationaddress  pickuplatd  pickuplong  driverlatd  driverlong


texts: Structured and unstructured pruning for deep learning models

texts: I was trying to understand structured and unstructured pruning techniques used for deep learning models: link 1 and link 2. To recap what I have understood that unstructured pruning is based on weight pruning however structured pruning is basically done through filter pruning, channel pruning, and filter shape pruning. So, my question is the unstructured pruning is done only for CNN models and cannot be done for example for MLP models as there are no filters and channels?


texts: Why did Logistic regression perform better than svm?

texts:
I have a data set of movies and their subtitles.My task is to classify them based on their ratings-[R,NR,PG,PG-13,G].
I have tried different ML algorithms and found that Logistic regression out performed them all, but I am unable to figure out why.My data had more features than observations.

SVM- should perform well on high dimensional data and will perform well even the there is class imbalance, but failed to show great results.
Naive Bayes-I think Naive Bayes did not perform well because of class imbalance.
Random forest-decent performance.but did not out perform logistic regression.

I am looking for an explanation as to why did one perform better than the other.

Note:The data set is sparse and it has more features/parameters than observations/examples.


texts: How to create clusters based on sentence similarity?

texts:
I have data which looks like following.
Data is a group of sentences which are similar, but have few unique words in between like TABLEA, TABLEB etc.

java.sql.SQLException: [Teradata Database] [TeraJDBC 15.10.00.22] [Error 3523] [SQLState 42000] The user does not have SELECT access to TABLEA
java.sql.SQLException: [Teradata Database] [TeraJDBC 15.10.00.22] [Error 3523] [SQLState 42000] The user does not have SELECT access to TABLEC
java.sql.SQLException: [Teradata Database] [TeraJDBC 15.10.00.22] [Error 3523] [SQLState 42000] The user does not have SELECT access to TABLEB
Dataframe read is null
Dataframe read is null
java.sql.SQLException: [Teradata Database] [TeraJDBC 15.10.00.22] [Error 3807] [SQLState 42S02] Object Y  does not exist.
java.sql.SQLException: [Teradata Database] [TeraJDBC 15.10.00.22] [Error 3807] [SQLState 42S02] Object Z  does not exist.
java.sql.SQLException: [Teradata Database] [TeraJDBC 15.10.00.22] [Error 2652] [SQLState HY000] Operation not allowed: TABLEK is being Loaded.
java.sql.SQLException: [Teradata Database] [TeraJDBC 15.10.00.22] [Error 9804] [SQLState HY000] Response Row size or Constant Row size overflow.
java.sql.SQLException: [Teradata JDBC Driver] [TeraJDBC 15.10.00.22] [Error 1000] [SQLState 08S01] Login failure for Connection to xxx.xx.xx.xx Tue Dec 04 02:49:47 MST 2018

Problem Statement:
I want to group/cluster the data and provide a unique number to each group/cluster.

Assumptions:

The groups/cluster should be formed based on the similarity. Similar sentences should be grouped in one cluster
This should be unsupervised learning. If in future, some new sentence comes which is very less similar to existing cluster, it should create a new group/cluster.
The sentences can be of any length
The common words between sentences can appear
anywhere - starting of string, in between, in the end, or so
The sequence of the words matter

Output:

The outcome should be a dimension table for category like below

Although I do get the problem statement in its abstract form, I do not know a concrete way to do this.

So far I have read about text clustering using various algorithms like cosine similarity and etc, but I am not sure if that will suffice this problem statement.
One of the major problem here is, it is unsupervised. If there are any new sentences whose similarity is very less then it should create a new group.

The bigger picture goes like this

Get list of all unclassified/uncategorised(I am using both interchangeably here since I am not sure which one it falls into) statements
Check in the dimension table, by matching using some similarity threshold(not clear on this).
If the similarity is matching above a threshold, then do nothing
If similarity is less, then create a new group in the dimension table with the Description column which has common words.

I have yet to identify what is the best approach to solve this problem.Please recommend some algorithm or approach to solve this problem.


texts: How can we calculate AUC for a simple decision tree?

texts: The setup is simple: binary classification using a simple decision tree, each node of the tree has a single threshold applied on a single feature. In general, building a ROC curve requires moving a decision threshold over different values and computing the effect of that change on the true positive rate and the false positives rate of predictions. What's that decision threshold in the case of a simple fixed decision tree?


texts: I am preparing classification model. Many of numeric variables are positives skewed. Should I change a distribution of variables to be more Gaussian?


texts: How to resample non-equidistant timeseries data in java?


texts: converting array to a true/false matrices

texts:
I have a data set where each record is a json document with a label, and an array of signals. The signals will vary for each record:
{
    "label":"bad",
    "id": "0009",
    "signals":["high_debt_ratio", "no_job"]
},

{
    "label":"good",
     "id": "0002",
    "signals":["high_debt_ratio", "great_credit", "no_id_match"]
},

{
    "label":"good",
    "id": "0003",
    "signals":["low_debt_ratio", "great_credit"]
},

{
    "label":"bad",
    "id": "0001",
    "signals":["high_risk_loc", high_debt_ratio", "no_job", "no_id_match"]
}

I want to convert this to a matrices that looks like this:

id
label
high_risk_loc
high_debt_ratio
no_job
great_credit
no_id_match
low_debt_ratio

0009
bad
false
true
true
false
false
false

0002
good
false
true
false
true
true
false

0003
good
false
false
false
true
false
true

0001
bad
true
true
true
false
true
false

I created a function but it seems like this would be a common thing to do. Is there a python lib (pandas, scikit, etc.) that does this for you? I'd rather use something from a package but i'm not sure what to search for.


texts: Survival Analysis: Pseudo Observation Vs Stratified Cox Regression. Which one is better?

texts:
I've been looking into the Cox Regression method for Survival Analysis in Churn Prediction. Cox regression will allow us to determine the probability that a subscriber will unsubscribe after a time $t$, defined by the hazard rate:

$$
h(t \lvert X_i ) = h_0(t)exp\big( \boldsymbol{\beta} ^T\boldsymbol{X}_{i} \big)
$$

Where

$h_0(t)$: Baseline Hazard is a prior Probability that any customer churns at time t when all influencing factors are 0.
$\boldsymbol{\beta} \in \mathbb{R}^D$: Exponent of each Coefficient gives us a Hazard ratio. These should be constant w.r.t time (proportionality assumption).
$\boldsymbol{X}\in \mathbb{R}^{N\times D}$: Set of $N$ sample customers

Problem: Proportionality Hazard Assumption: Cox regression makes an assumption that the Hazard Ratios should remain constant through time $t$. For example, for a covariate $X_1$ = "gender", say $\beta_1=1.8$. In english, it means male subscribers tend to leave the service $80\%$ more than females after a time $t$. However, this $80\%$ should hold for any time $t$.

This is usually an unreasonable constrain for many variables. But there are other methods that can incorporate variables that don’t follow the proportional hazards assumption.

stratified cox regression
pseudo-observations
cox regression with time-dependent covariates

I was just reading up on stratified cox regression. The only apparent downside here is:

The variables that are stratified need to be converted into categorical variables
The stratified categorical variables should not have too many degrees of freedom. This will lead to a LARGE number of models whose parameters need to be estimated.

Question: Is pseudo-observations similar? Does it have less/more rigid constraints? Even so, how is it's performance considering I have copious amounts of data?


texts: Mean error (not squared) in scikit-learn cross_val_score

texts: I need to know if the values generated by each fold of cross_val_score have a distribution which is centered on zero.  Something as simple as the median or mean of y_true - y_predicted would suffice.  All I see in the available options are absolute and squared.  I've looked into make scorer but can't see how to code the simple mean error and then call it as the scoring argument in cross_val_score.


texts: Valid approach? LogSoftmax during training, Softmax during inference

texts:
I am training a classifier assigning one of four possible classes to each frame in a preprocessed audio stream using pytorch. I am using cross-entropy loss as the loss function for training. It is implemented as a combination of LogSoftmax and negative log-likelihood loss. During inference however I am using a softmax layer before the output, omitting the logarithm. Now I am wondering whether this yields any problems I might overlook. I suppose it should be fine, because in the end I'm deciding to choose one class and as both softmax and log-softmax are monotonic, the highest probability should belong to the same class using either. The reason I'm not simply switching to log-softmax during inference as I would loose the desirable effect of the output being an actual probability distribution if I'm not mistaken.
Are my assumptions correct so I may carry on or did I miss something important?


texts: How to annotate text documents with meta-data?

texts:
Having a lot of text documents (in natural language, unstructured), what are the possible ways of annotating them with some semantic meta-data? For example, consider a short document:

I saw the company's manager last day.

To be able to extract information from it, it must be annotated with additional data to be less ambiguous. The process of finding such meta-data is not in question, so assume it is done manually. The question is how to store these data in a way that further analysis on it can be done more conveniently/efficiently?

A possible approach is to use XML tags (see below), but it seems too verbose, and maybe there are better approaches/guidelines for storing such meta-data on text documents.

<Person name="John">I</Person> saw the <Organization name="ACME">company</Organization>'s
manager <Time value="2014-5-29">last day</Time>.


texts:
I am completely new to data science and looking to narrow down the search and reduce the learning curve required to solve problems like the one given below

I have a data set with 7 columns ,
Column A(all positive decimal) is the data point I want to maximize.
Column B and C are boolean values
remaining columns are a combination of positive and negative decimal numbers.
I want to find some relation and insights from all colums such that I can maximize the sum of column A.


texts: Is there a replacement for small p-values in big data?

texts: If small p-values are plentiful in big data, what is a comparable replacement for p-values in data with million of samples?


texts: can I say that my variable is "approximately" normally distributed?

texts:
I am studying a variable on 50000 observations, I have applied a cox box transformation to make it normal, but even with transformation the kolmogorov smirnof and the anderson darling test tells that my distribution is not normal,
given the qqplot that I have, and the number of observations can I say that my variable is approximately normal? I want to use parametric test but all the test rely on the normality assumption


texts: Different approaches of creating the test set

texts:
I came across different approaches to creating a test set. Theoretically, it's quite simple, just pick some instances randomly, typically 20% of the dataset and set them aside. Below are the approaches

  The naive way of creating the test set is

def split_train_test(data,test_set_ratio):
  #create indices
  shuffled_indices = np.random.permutation(len(data))
  test_set_size = int(len(data) * test_set_ratio)
  test_set_indices = shuffled_indices[:test_set_size]
  train_set_indices = shuffled_indices[test_set_size:]
  return data.iloc[train_set_indices],data.iloc[test_set_indices]

The above splitting mechanism works, but if the program is run, again and again, it will generate a different dataset. Over the time, the machine learning algorithm will get to see all the examples. The solutions to fix the above problem was (guided by the author of the book)

Save the test set on the first run and then load it in subsequent runs
To set the random number generator's seed(np.random.seed(42)) before calling np.random.permutation() so that it always generates the same shuffled indices

But both the above solutions break when we fetch the next updated dataset. I am not still clear with this statement.

  Can someone give me an intuition behind how do the above two solutions breaks when we fetch the next updated dataset ?.

Then the author came up with another reliable approach to create the test.

 def split_train_test_by_id(data, test_ratio, id_column):
   ids = data[id_column]
   in_test_set = ids.apply(lambda id_: test_set_check(id_, test_ratio))
   return data.loc[~in_test_set], data.loc[in_test_set]

  Approach #1

 def test_set_check(identifier, test_ratio, hash=hashlib.md5):
    return bytearray(hash(np.int64(identifier)).digest())[-1] < 256 * test_ratio

  Approach #2

 def test_set_check(identifier, test_ratio):
    return crc32(np.int64(identifier)) & 0xffffffff < test_ratio * 2**32

  Approaches #1,#2, why are we making use of crc32, 0xffffffff, byte array?.

Just out of curiosity, I passed different values for identifier variable into hash_function(np.int64(identifier)).digest() and I got different results.

  Is there any intuition behind these results ?.


texts: I have around 40K question and answers. How can I build a machine learning model so that if any new question comes it has to detect as duplicate or not?


texts: Convolutional neural network with cross validation in Keras

texts:
I want to use K-fold cross-validation on my dataset of images. I am reading the data (images) from a directory.

How do I use cross validation with convolutional neural network in Keras?


texts: CheXnet model training and validation loss stuck

texts:
I am training a CheXnet like model from scratch. I followed paper (https://arxiv.org/pdf/1711.05225.pdf) and built the model similar. But I am unable to train it, during end-to-end training the training and validation loss get stuck at 0.1325 and
0.1489 respectively.
Current model specifications:

DenseNet121 (+ FC layer of 14 units), ImageNet weights, end-to-end
training
Unweighted binary crossentropy loss
Adam optimiser(lr=0.001), Learning rate decay of 0.1 if validation loss did not
improve for 2 continuous epochs

I have tried changing the batch size but no luck. Can someone please help me figure out what should I change.
Thanks.


texts: R Programming, how to replicate for districts in a city

texts:
I am new to R Programming and just learned basics through codeschool.com

Our network spans the city, and it is divided into districts.

I would like to create a map that assigns a value (based on ratio of outdated software and new software) to each district.

This website has sample of 3-D maps that were created by R Programming, and I see one I am very interested in replicating, but for our city only.

But when I see the source code, I don't see any mention of latitude or longitude.

My head is spinning, trying to figure out how I will input this, i.e latitude and longitude of a district in our city, versus an assigned ratio, which I believe will be read from a spreadsheet.

Thanks for any guidance.


texts:
I am doing a small project with the basic idea of recognizing a face. I made a version that uses face_recognition library. And, although I am quite satisfied with the results I am not happy with the time it takes for the program to finish. I am quite sure that this is due to face_recognition does not use GPU/CUDA despite me having GPU on my machine. in particular the function

face_recognition.face_locations(picture, model="cnn")

does all the computation on CPU. In the docs of this function it says that it will use CPU if it cannot use GPU. However, I used GPU with another library and it was fine. I would be happy for any help that you can give me.

P.S. Probably you know other libraries that I probably should pay attention to.

Edit: the link to the face_recognition library.


texts:
I have been searching for 2 weeks and I got no where so far.

There is  a list of diseases

Diabetes
Heart Transplant
Fingertip amputation
Injury by sharp tools
.
.
.

and My dataset is a list of medical text reports.

the training dataset has diseases that can be generated from each record

example that I made

  This patient has suffered a cut while using his Carving Chisel and led
  to losing the fingertip therefore we had to operate to sew
  the tip.....

from this report we get these diseases

1- Injury by sharp tools
2- Fingertip amputation
3- Sewing injury

another report results may have 3 or less or more diseases

I have searched a lot
I found many examples about NLP classification

where a text will be classified into Sports, Politics, Culture, Science, etc.

I found NER where person names, locations, dates, etc can be extracted from a text.

But did not find anything for a single text could have multiple values (similar to my dataset)

I dont know where to start.

Could anyone please help me finding what is the name of this method of extracting list of issues from a text?

Edit

What else do I need to exclude the negations, if the report says

  "This patient has stomach problem but not diabetes "

How can I make AI understand there is a negation (NOT) before diabetes so it should not be included?

So the result will be

stomach problem

as diabetes should be excluded because of the negation word


texts: Trouble with Scrapy and Python: ever present unexpected indent

texts:
I am trying to build a scrapy bot capable of ripping the data from my local craigslist for jobs as well as having a recursive functionality to allow for the contact data to be gathered as well. Ultimately I would like to have all of this data placed into a .CVS file. I have read Learning Python the Hard Way as well as a good portion of Automating the hard stuff with Python, however I am still quite a novice. I have referenced the following tutorials while trying to build this script:
the documentation for Scrapy & this blog post by Michael Herman. I feel as though my code simply has a dumb syntax error but I'm too ignorant of what to do to fix it.

Here's the code for the spider:

from scrapy.contrib.spiders import CrawlSpider, Rule
from scrapy.contrib.linkextractors.sgml import SgmlLinkExtractor
from scrapy.selector import HtmlXPathSelector
from craigslist_sample.items import CraigslistSampleItem

class MySpider(CrawlSpider):
name = "craigs"
allowed_domains = ["sfbay.craigslist.org"]
start_urls = ["http://salem.craigslist.org/search/jjj"]
#Initially grab all of the urls up to where craigslist allows
#In this case, it's 2400
for i in range(1, 24):
    start_urls.append(base_url + "s=" + str(i) + "00&")

rules = (
    Rule(SgmlLinkExtractor(allow=(), restrict_xpaths=('//a[@class="button next"]',)), callback="parse_items", follow= True),
)

def parse_items(self, response):
    hxs = HtmlXPathSelector(response)
    titles = hxs.xpath('//span[@class="pl"]')
    items = []
    for titles in titles:
        item = CraigslistSampleItem()
        item["title"] = titles.xpath("a/text()").extract()
        item["link"] = titles.xpath("a/@href").extract()
        item["email"] = titles.xpath("@anonemail").extract()
        items.append(item)
    return(items)

The following is the exact error I am receiving:

*File "C:\Users\newfa\Documents\scripts\craigslist_sample\craigslist_sample\spiders\test.py", line 27
item["email"] = titles.xpath("@anonemail").extract()
^

IndentationError: unexpected indent*

I do understand that indentation is very important in Python, however I keep trying different methods of indentation and have tried several "beautify" code methods to try and get it right, which leads me to believe that it may be some other error.


texts: Pre-processing (center, scale, impute) among training sets (different forms) and the test set - what is a good approach?

texts:
I am currently working on a multi-class classification problem with a large training set. However, it has some specific characteristics, which induced me to experiment with it, resulting in few versions of the training set (as a result of re-sampling, removing observations, etc).

I want to perform pre-processing of the data, that is to scale, center and impute (not much imputation though) values. This is the point where I've started to get confused.

I've been taught that you should always pre-process the test set in the same way you've pre-processed the training set, that is (for scaling and centering) to measure the mean and standard deviation on the training set and apply those values to the test set. This seems reasonably to me.

But what to do in case when you have shrinked/resampled training set? Should one focus on characteristics of the data that is actually feeding the model (that is what would 'train' function in R's caret package suggest, as you can put the pre-processing object in there directly) and apply these to the test set, or maybe one should capture the real characteristics of the data (from the whole untouched training set) and apply these? If the second option is better, maybe it would be worth it to capture the characteristics of the data by merging the training and test data together just for pre-processing step to get as accurate estimates as possible (I've actually never heard of anyone doing that though)?

I know I can simply test some of the approaches specified here, and I surely will, but are there any suggestions based on theory or your intuition/experience on how to tackle this problem?

I also have one additional and optional question. Does it make sense to center but NOT scale the data (or the other way around) in any case? Can anyone present any example where that approach would be reasonable?

Thank you very much in advance.


texts:
from nltk.chunk import ChunkParserI
from nltk.chunk.util import conlltags2tree
from nltk.corpus import gazetteers

class LocationChunker(ChunkParserI):
    def __init__(self):
        self.locations = set(gazetteers.words())
        self.lookahead = 0
        for loc in self.locations:
            nwords = loc.count(' ')
        if nwords > self.lookahead:
            self.lookahead = nwords

What is ChunkParserI  in nltk.chunk ? What exactly it has been called for?
Also, please explain the code.
What is the difference between chunking and parsing?


texts:
I would like to grab tensor v1's third dim according to indices in tensor indices. I tried torch.take and torch.gather, but no go. See example:

import torch
v1 = torch.randn((1,35,20,200))
indices = torch.randint(0, 20, (1,35,20))
out = torch.gather(v1, 2, indices)

This gives an error as v1 and incides dimensions don't match. Out should be of shape [1,35, 20, 200]. How would I do that in PyTorch?


texts: Anomaly Detection Methods for Clean Training Data

texts:
The goal is to find anomalies in my dataset, univariant anomaly detection which basically means looking for anomalies only in one column.

Up until this point I have tried DBSCAN; Isolation Forest and K-Means.

Recently I have been told that if the training data is does not have anomalies I can use autoencoders and other methods.

I am a bit confused on what are the other methods that can be used in cases where your training data has no anomalies. You do the training on them and then you run the model on the Test Data and check how good your algorithm is working.

I have been trying to find articles and papers regarding this but most of them don’t talk or say if the training data does or doesn’t have anomalies.


texts: Time Series Analysis / Modeling with auto_arima

texts:
I recently dived into Time Series and was attempting at doing some data analysis and modeling. I'm using the following dataframe
             meantemp   humidity  wind_speed  meanpressure
date
2013-01-01  10.000000  84.500000    0.000000   1015.666667
2013-01-02   7.400000  92.000000    2.980000   1017.800000
2013-01-03   7.166667  87.000000    4.633333   1018.666667
2013-01-04   8.666667  71.333333    1.233333   1017.166667
2013-01-05   6.000000  86.833333    3.700000   1016.500000
...               ...        ...         ...           ...
2017-04-20  34.500000  27.500000    5.562500    998.625000
2017-04-21  34.250000  39.375000    6.962500    999.875000
2017-04-22  32.900000  40.900000    8.890000   1001.600000
2017-04-23  32.875000  27.500000    9.962500   1002.125000
2017-04-24  32.000000  27.142857   12.157143   1004.142857

[1575 rows x 4 columns]

describing the daily mean temperature, humidity and other stuff, but I'm focusing mostly on the 'meantemp' feature.
The plot looks like this:

I had the intuition that this time series might be stationary, so I ran aan augmented Dickey-fuller test to check it..
dftest = adfuller(df['meantemp'])
dfout = pd.Series(dftest[0:4], index=['ADF','P-value', 'Usedlags', 'nobs'])
dfout

and I obtained
ADF           -2.377592
P-value        0.148161
Usedlags      10.000000
nobs        1564.000000
dtype: float64

So since p-value is >0.05 I can't assume it is actually stationary. At this point I ran an ETS
decomp = seasonal_decompose(df['meantemp'], period =12)
decomp.plot();

Now I already have some perplexities, since the seasonality component is pretty 'small' compared to the range of  the data..(should I ignore the seasonality?) Anyway then I use auto_arima to look for an appropriate model
auto_arima(df['meantemp'], seasonal=True, m=12).summary()

Now I have question.. What's the meaning in auto_arima returning back a SARIMAX model with just three components? Are those the 'order' components or 'seasonal' components? I'm pretty sure they are the 'order' components but then why doesn't it return back just an ARIMA one?
Also when I fit the model and predict on test data I obtain the following plot:

I also checked the RMSE and it's about 6.(numbers) compared to the mean of the whole data that is about 25... Does all of this make sense? Thank you!


texts: Implementing sklearn's FeatureHasher on Unseen Data

texts:
For a little bit of background I have been working on a binary classification of health insurance claims and am implementing sklearn's FeatureHasher to vectorize categorical features, many of which are particularly high in cardinality with a high count of unique factor levels and sklearn's FeatureHasher has been a useful tool to encode all of this information.

I have trained an XGBoost classifier on my initial training set which was transformed such that each categorical feature was hashed into log(x) dimensions (x being the number of unique factor levels within in each feature).  I am planning to use the trained model to make predictions on new/unseen claims data which will be scored on a daily basis.  I will be hashing the daily extract of claims data with identical hashing dimensions used in the training set, such that there are no disparities between the number of features the model was trained upon and the number of features within the daily extract, allowing me to use the trained model to make predictions on incoming claims data.

I am wondering how sklearn's FeatureHasher encodes unseen data.  For example, if I had ten factor levels in a specific categorical feature within my original training set and only three factor levels within the same categorical feature in an incoming daily extract of claims data (likely to be a much lower volume of claims data as compared to the training set), does FeatureHasher encode this information in the same way such that each factor level can be identified and mapped back to its original value?  Conversely, will hashing encode factor levels within categorical features differently between training and test sets if the number of factor levels within a given feature are not equal to what the model was trained upon? If so would this cause issues with how the model interprets the factor levels of a given categorical feature within a daily extract of claims causing the model to confuse or misinterpret factor levels from how they were originally encoded in the training set?

Any feedback would be greatly appreciated!

P.S. I have looked into using CatBoost gradient boosting (catboost.ai/docs/concepts/python-installation.html) as an alternative but unfortunately am using a platform that does not support this and thus will not be an option for me


texts: Getting value of 2 in normalized confusion matrix

texts:
I have created confusion matrix with python and I have gotten weird results that made me worried:

as you can see, I have value of two, and if I understand correct, normalized matrix cannot have value higher than 1. and also, some columns' sum is not 1.
this is how I created this matrix:
df_confusion = pd.crosstab(df['labels'], df['prediction'])
df_conf_norm = df_confusion / df_confusion.sum(axis=1)

and then I just plot it.
my question is - is it possible to get value of 2? does the sum of column make sense not to be 1? do I have mistake? and if correct, what can explain that?
EDIT


texts: (Deep Learning) Backpropagation derivation from notes by Andrew NG

texts:
I am self-studying Andrew NG's deep learning course materials from the mcahine learning course (CS 229) of Stanford. The material is available here.
I have a question about the chain rule techniques used in deriving the backpropagation step shown below (Equation 3.28 on page 12). Specifically, I wonder how come $\frac{\partial z^{[2]}}{\partial W^{[2]}}=a^{[1]}$? Obviously, the result couldn't be a vector (i.e., $a^{[1]}$) if we differentiate a vector ($z^{[2]}$) with respect to a matrix ($W^{[2]}$).

Also, the notes subsequently says that the size of the two sides do not match up. This really confuses me: if the derivation is correct, then how could the sizes of both sides not equal (see below)?
I will greatly appreciate if anyone can help explain the steps here! I spent many days and nights already but have no progress at all. Thanks!


texts: Cost-complexity pruning: what the method does exactly?

texts:
The tree::prune.tree R function has a method parameter, described in the guide as:

character string denoting the measure of node heterogeneity used to guide cost-complexity pruning. For regression trees, only the default, deviance, is accepted. For classification trees, the default is deviance and the alternative is misclass (number of misclassifications or total loss).

For regressions, method must be deviance, for classification, it can be deviance or misclass.
In The Elements of Statistical Learning (2nd ed), cost-complexity pruning is described for regression trees at pag. 308.
Given the full tree T_0, for every $\alpha$, we find the sub-tree $T_\alpha \subseteq T_0$ that minimizes $C_{\alpha}(T)$.
$$C_{\alpha}(T) = \sum_{m=1}^{|T|} \sum_{x_i \in R_m} (y_i - \hat{c}_m)^2 + \alpha |T|$$
Where is the deviance of the method parameter of the tree::prune.tree R function?
In case of classification tree, how should we modify $C_{\alpha}(T)$ in case we want to use the deviance method?
What in case we want to use the misclass method?


texts: What is the difference between observation and variable?

texts:
I have a matrix with size m×n that is built from n number of individuals for person identification. So, n is the number of person and m is the number of feature's value for the person.

It makes me confused about observation and variables. What will I call n and m? Which one represents observation and which one represents variable? My confusion will be cleared to you if you visit the following link: How to do SVD and PCA with big data?


texts: Input data from time series, keeping part of the steps to predict

texts:
I am using Keras LSTM for time series forecasting.

I am facing an issue, with transforming my data correctly.

I have a big dataset with values as follows
[[x1,x2,x3,x4,x5,x6,x7,x8],
[x1,x2,x3,x4,x5,x6,x7,x8],
[x1,x2,x3,x4,x5,x6,x7,x8]....]

I want to predict x8.
x1-x7 are values that are available before x8 is found, and x8 is dependant upon them.

Ideally, as the input into the LSTM, I'd have say 10 rows of all variables and then 5 rows of x1-x7, with the 5 rows of x8 as output, but I am having troubles getting such a vector into it. How should I proceed?


texts:
I've never understood how to calculate an autoencoder loss function because the prediction has many dimensions, and I always thought that a loss function had to output a single number / scalar estimate for a given record. However, on GitHub I recently came across a repository that has implemented an example autoencoder in TensorFlow, and the squared error implementation is a lot simpler than I thought:

cost = tf.reduce_mean(tf.pow(y_true - y_pred, 2))

The TensorFlow documentation on reduce_mean says, among other things:

  If axis has no entries, all dimensions are reduced, and a tensor with
  a single element is returned.

Can I conclude from all of this that the squared error of an autoencoder prediction is just the average across all of the record's dimensions?


texts: Detecting person on the picture if we know it is there

texts: I implemented an algorithm for pedestrian classification using Keras and Tensorflow and it is quite good, it has 96 percent accuracy on test data. It gives you information about existence of a person on the picture. Now I want to use that to detect where the person is. How to do that? For example, I want to build rectangle around the pedestrian but I don't know how to do that if I already know that there is a person on the picture.


texts: With regards to VC-dimension, why can you shatter 3 points with circles but not 4 points?

texts:
When using VC-dimensions to estimate the capability of a binary classifier, you can find 3 points in R2 that can be shattered, e.g.:

But you can not shatter any 4 points with a circle.

This is stated in these lecture notes. Can anyone give me an intuitive explanation?


texts: Feature selection, is it possible to combine wrapper and embedded methods?

texts:
I'm using neural network to predict PM10 concentration (a regression problem). Since the wrapper method is dependent on the model, so passing the neural network model that's optimized for all the features in the dataset won't work correctly (since every time the wrapper reduces the features in the dataset, the hidden layer size of the neural network must be re-tuned).

So my question is, can I use the embedded method (Lasso) as an estimator in the wrapper method?

This is the feature selection part that I' asking about:

from mlxtend.feature_selection import SequentialFeatureSelector

feature_selector = SequentialFeatureSelector(LassoCV(),
           k_features=3,
           forward=True,
           verbose=0,
           scoring='neg_mean_absolute_error',
           cv=0)
features = feature_selector.fit(X, y)
filtered_features= cols[list(features.k_feature_idx_)]
print(filtered_features)

I'm new to data science and machine learning, so I want to know if this is right or wrong.


texts: Weighting training data from time-series

texts:
I want to evaluate whether or not it is feasible for me to train a machine learning model using sample data that I can acquire.  I have many time-series, and I am able to establish groups of 'related' time series - some groups are more similar than others.  My hypothesis is that the history of the related time-series contains sufficient information to make some predictions about a particular time-series. I would like to make the most accurate predictions possible.
Assume I were to want to make predictions about time-series A, which is part of a group of related time series - eg. {A,B,C,D,E} - then the most relevant training data might be historic time-series for A,B,C,D and E... with the most recent examples being more relevant than the eldest.  Similarly, a different time-series Z, in another group of related time-series - e.g. {Z,Y,X,W,V} might exhibit patterns relevant to prediction... but, by virtue of distinct things being measured, perhaps these examples are lower quality (from the perspective of making predictions about A) than the examples drawn from actual measurements of A and its related time-series.  Perhaps there's a cross over where more recent data about some other variables might be more relevant than elder data about the same variables.
I would like to know if there are any tools/frameworks that would help me train a machine learning model using sample data where the relevance of each training example is not uniform. Can anyone point me at a book, paper, or blog that tackles this practical problem?


texts: cosine_similarity returns matrix instead of single value

texts:
I am using below code to compute cosine similarity between the 2 vectors. It returns a matrix instead of a single value 0.8660254.

  [[ 1.         0.8660254]

  [ 0.8660254  1.       ]]

from sklearn.metrics.pairwise import cosine_similarity
vec1 = [1,1,0,1,1]
vec2 = [0,1,0,1,1]
print(cosine_similarity([vec1, vec2]))


texts: Why there is two output in Titanic case in tflearn quickstart?

texts:
I am stuck in the tflearn quickstart guide.

I've tried tit and it works perfectly with accuracy around 78%, but the problem is I don't understand why the "labels" consist of two "survived" columns that are the opposite of each other. I've trained the same data but used only single "survived" column in the labels or the output, but with the same code I just get around 36% with 0 loss or even NaN loss.

So why is there two columns output/labels? And why can't I use only a single column?


texts: Search n-grams in a text against a list of n-grams

texts:
So, I need to perform word extraction from a python string by matching it against a list of tokens that can have bigrams as well as unigrams, but this can go to higher order n-grams. Expected size of list of tokens is 0.2 million.
My approach for another list of tokens that is way smaller than the 0.2 million list is to perform tokenisation on the text after cleaning the text and removing stop words.
What could be the best approach to search for unigrams and bigrams in a 0.2million index. Intersection method would be very inefficient on this huge index.
I have written a code using Spacy's Matcher to search for a small set of bigrams in the text.
This is my code for TOKEN intersection.
TOKENS = ['contract','permanent','temporary','longterm','shortterm']

def token_intersection(text):
    jt = []
    global TOKENS
    try:
        tokens = nltk.word_tokenize(text.lower())
        jt.extend(list(set(tokens).intersection(TOKENS)))
    except Exception as e:
        print("Exception at job_type",e)
    return jt

And for finding a specific bigram the code is as below:

def match_pattern(self,bigram:list):
        text = self.text.lower()
        try:
            import spacy
            from spacy.matcher import Matcher
            nlp = spacy.load("en_core_web_sm")
            matcher = Matcher(nlp.vocab)
            pattern = [{"LOWER": bigram[0]}, {"LOWER": bigram[1]}]
            matcher.add("matchID", None, pattern)
            doc = nlp(text)
            matches = matcher(doc)
            found = len(matches)
            if found > 0:
                return "".join(bigram)
            else:
                return None
        except Exception as e:
            print("Error occured at matching pattern",e)

If I use the token_intersection method on 0.2million list of tokens, I will miss bigrams, and if I use the match_pattern method, I will have to iterate over all the 0.2million elements.
How can this be done better, fast and efficiently?


texts: Custom Object Detection on Azure Machine Learning or Databricks

texts:
I'm looking to train a custom object detection model using Tensorflow's API. Is it possible to do it in Azure ML Studio or in Databricks?

Object Detection link - https://tensorflow-object-detection-api-tutorial.readthedocs.io/en/latest/training.html

Can this be setup without using COCO?


texts: What are some function/package in R to find similarity of individual words not in the context of sentences?

texts: what are some function/package in R/python to find similarity of individual words not in the context of sentences? As a novice, I searched and found methods like adist, cosine similarity, word2vec, but not sure what are the supposed use case for each. Appreciate any advice.


texts: Question About Discriminator of CycleGan

texts:
The Discriminator of CycleGan outputs not just a single value to say that the image is real or fake.... But It outputs a grid of numbers (like 8X8 or 7x7), where each number says whether one patch of the input image is fake or real.
So, my question is that, why do we do this.. What benefits does it gives us, and what was the problem in the approach where we only outputted a single value?


texts: How do we know a neural network test accuracy is good enough when results vary with different runs?

texts:
In every paper I read about prediction models, the training accuracy and the test accuracy (sometimes also the validation accuracy) is stated as a discrete number. However, in experience, depending on how the weights are initialized, different training runs result in different testing results.
How does a standard data science researcher pinpoint the accuracy metric to be written in a paper, and how does ze get certain about its validness?


texts: Check similarity of table/csv of Product Names

texts:
We've got a list of approximately 18,000 product names (they're from 80-90 sources, so quite a few that are similar but not duplicates - these were picked as DISTINCT from a table) unfortunately there are different ways of expressing these names. We have to try and normalize the dataset so we present our users with more meaningful names.

For example, a list like this:

Canon EOS 5D Mark III
Canon EOS 5D mk III
Canon EOS 5DMK3
Canon EF 70-200mm f/2.8L IS II USM Lens
Canon EF 70-200mm f/2.8L IS II USM Telephoto Zoom Lens
Canon EF 70-200mm f/2.8L IS USM Lens
Canon EF 70-200mm f/4L USM Lens

I'd like to be able to assess those strings and collapse them into something like this:

Canon EOS 5D Mark III
Canon EF 70-200mm f/2.8L IS II USM
Canon EF 70-200mm f/2.8L IS USM Lens
Canon EF 70-200mm f/4L USM Lens

But I'd like to know how similar two strings are to be able to determine this. I do realise that the F2.8 IS II and IS USM may be a bit hard, but thought I'd throw it in.

The real product names are far less exciting (they're parts for the farm equipment we stock).

We also store these names in a Postgres (9.5) database table. Examples i've seen compare two lists, but we don't have a master product list to do that unfortunately.


texts:
I'm new to the field of time series prediction.
I'm looking for a demand prediction model to predict when the product will be sold out from the online supermarket (when the supply is known in advance).

I know that LSTM and ARIMA are the "best" model for time series prediction. Is there any other model for demand prediction which could be better?

and what are the con and pros for those model for demand prediction?

Also, can someone give me some time frames to developing this kind of model? due I need to consider anything else than things in the regular model (like hyper-parameters tuning and etc.)


texts: Overfitting in Huggingface's TFBertForSequenceClassification

texts:
I'm using Huggingface's TFBertForSequenceClassification for multilabel tweets classification. During training the model archives good accuracy, but the validation accuracy is poor. I've tried to solve the overfitting using some dropout but the performance is still poor. The model is as follows:
# Get and configure the BERT model
config = BertConfig.from_pretrained("bert-base-uncased", hidden_dropout_prob=0.5, num_labels=13)
bert_model = TFBertForSequenceClassification.from_pretrained("bert-base-uncased", config=config)

optimizer = tf.keras.optimizers.Adam(learning_rate=3e-5, epsilon=0.00015, clipnorm=0.01)
loss = tf.keras.losses.CategoricalCrossentropy(from_logits=True)
metric = tf.keras.metrics.CategoricalAccuracy('accuracy')

bert_model.compile(optimizer=optimizer, loss=loss, metrics=[metric])
bert_model.summary()

The summary is as follows:

When I fit the model, the outcome is:
history = bert_model.fit(train_ds, epochs=30, validation_data = test_ds)


texts: Difference between business and financial and data analyst

texts:
Can anyone please tell me whats the difference between

Business analyst
Financial analyst
Data analyst

And what should one need to learn in order to achieve either of these profile.


texts: Are there decisive leaders in programming with tabular data?

texts:
What are the most effective bread-and-butter in-memory open source tabular data frameworks today? I have been working with tabular data for years with an in-house solution that integrates with Excel well, but falls short of many other expectations. I would like to (if possible/true) demonstrate that our solution has fallen behind the times.

In other words, assuming an SQL-like platform is responsible for persistence of a data set, but cycle intensive calculations need to be performed on that dataset (E.g. stochastic simulation processes), an efficient framework to program in is . Advantageous features, to give an idea

efficient use of memory for common operations like random access, sorting, map/reduce/filter.
performant when serializing and deserializing
offers good expressiveness or extensibility
Isn't bound by oppressive commercial licensing agreements

The dataframe of pandas is the best product I can find in the community, but it is a mixed bag as far as performance is concerned. Matlab comes to mind but is excessively commercialized to the point that using it for most distributed applications on homegrown cloud application becomes a nightmare.


texts: Apply error analysis on the iris dataset for a specific type of misclassification

texts:
Suppose that I have the well-known iris dataset and I want to perform error analysis on the misclassified examples, more specifically for a specific class.
I don't really care about fine-tuning or selecting another model, stratification when splitting etc. which would improve precision and recall for each class, rather than how to manually focus on the features of the misclassified examples and understand the logic of misclassification (if you could please avoid being too technical e.g. using packages as lime or shap).
Below I provide an MWE to make it more clear.
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import seaborn as sns

from sklearn import datasets

from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report, confusion_matrix

from sklearn.linear_model import SGDClassifier

Get the data and train a model.
iris = datasets.load_iris()

X = iris.data
y = iris.target_names[iris.target]

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.3, random_state=42
)

sgd = SGDClassifier(random_state=42)
sgd.fit(X_train, y_train)
y_pred = sgd.predict(X_train)

conf_matrix = confusion_matrix(y_true=y_train, y_pred=y_pred)
conf_matrix

The confusion matrix will look like:
array([[31,  0,  0],
       [ 1, 17, 19],
       [ 0,  0, 37]], dtype=int64)

and the classification report will be like:
              precision    recall  f1-score   support

      setosa       0.97      1.00      0.98        31
  versicolor       1.00      0.46      0.63        37
   virginica       0.66      1.00      0.80        37

    accuracy                           0.81       105
   macro avg       0.88      0.82      0.80       105
weighted avg       0.87      0.81      0.79       105

Looking at the classification report we see that the recall of the versicolor class is really low and actually we have more false negatives (in favor of the virginica class), rather than true positives (skewed classes is not the case here). Also, notice that within the code, we test the performance in the training data.
My feeling says that the examples where we have misclassification y_true=versicolor and y_pred=virginica are more "similar" to the examples where we have y_true=virginica, but I am not quite sure how to measure this similarity.
Would this error analysis be different if we chose a different model, e.g. a RandomForest?
In reality, we would have potentially a lot of features, like one-hot encoded ones, text along with numerical ones, but I gave this MWE as a start on how to proceed.


texts: Right Way to Input Text Data in Keras Auto Encoder

texts:
I have several thousand text documents and I am currently working on obtaining the latent feature representations of words and generate sentences using variational auto encoder. The main obstacle I am facing is “how to input such large textual vectors into VAE (or say even in a simple auto encoder)". I know there are other techniques such as adding layers in the form of LSTMs, but I want to keep it simple and keep the model as just VAE. One way is to use one-hot-encoded vectors or bag of words, but again, this is not the most efficient way since for a vocabulary of 100K unique words, each document will have a 100K input vector. Additionally, we loose the sentence structure. For small datasets however, there is no problem in training an autoencoder using this type of input. Another method is to use word embeddings using a pre-trained Word2Vec. This is what I have been trying to do and the python notebook which can be DOWNLOADED HERE uses this technique. The code is too long and has multiple pre-processing steps, so I am unable to embed this code in my post. The following are my questions:

Now, each sentence (or document) will have different number of words. So the number of word embeddings for each  document will have different lengths. Unfortunately, keras requires all the inputs to be of same length ( if I am right). So, how to handle such cases of varying input lengths?. Currently, in the fifth block of the python notebook, you can see the statement data = [x for x in vect if len(x) == 10]. That is, I only consider documents that have exactly 10 words to overcome this problem. Ofcourse, this is not practical. Can we pad 0 vectors?
The VAE example shown in Keras blog uses MNIST data as exmple. Therefore, they use a sigmoid activation in the final reconstruction layer; consequently, “binary_crossentropy” as the loss function (along with the KL divergence). Since my inputs are word embeddings, where there are even negative values in the embeded vectors, I believe I should not use the activation as “sigmoid” in the final reconstruction layer.  Is that right?. Additionally, I have also changed the loss as  “mean_squared_error” instead of “binary_crossentropy” in the attached code.

It will be great if someone who has worked on VAE and autoencoders for text data can provide their inputs regarding the questions mentioned above.

Note: the attached code is a simplified version of the code in THIS LINK


texts:
I am new to word/paragraph embedding and I am still trying to understand it. My question may be trivial. So please bear with me.

I have a collection of short documents (each documents contain few sentences and lots of noise). I want to cluster these documents according to the most similar documents into one cluster (soft cluster is fine for now). Now there are several techniques available (and noted tutorials such as in scikit-learn) but I would like to see if I can successfully use doc2vec (gensim implementation).

To this extent, I have ran the doc2vec on the collection and I have the "paragraph vector"s for each document. Obviously, I can cluster these vectors using something like K-Means. I am just wondering if this is the right approach or there is something else is needed.

Thanks


texts: General Machine Learning Workflow Question

texts:
I'm very new to machine learning and want to understand the general process by which it is carried out. I've worked through the famous 'iris' tutorial and want to ask if the principles in that tutorial are applicable to every future machine learning project I undertake.

I work in biological sciences, and am interested in applying machine learning algorithms to biological sequence data as away of categorizing classes or doing unsupervised clustering. From my understanding, each project starts with:

Me defining the 'scope' or aim of what I want to learn from the raw data.
Generating features/attributes from my OWN code/algorithms that could potentially differentiate between the two classes
This basically generates a huge matrix of x_features by y_entries
Feed the matrix into a machine learning algorithm (I'm sure this is vastly oversimplified).

To give an example, say I have 10,000 protein sequences, and I believe 5000 are 'Class1' and 5000 are 'Class2', but I do not know how to differentiate them by eye. I need to generate x_features (in some informed way) of this sequence using my own custom algorithms, and feed the resulting 10000 entries into an algorithm.

Is this the right approach? I'd be eternally grateful if someone could direct me towards a beginners tutorial that revolves around analyzing biological sequence data.


texts: Bert for QuestionAnswering input exceeds 512

texts:
I'm training Bert on question answering (in Spanish) and i have a large context, only the context exceeds 512, the total question + context is 10k, i found that longformer is bert like for long document, but there's no pretrained in spanish so, is there any idea get around bert.
What i tried is:
from transformers import BertConfig
config=BertConfig.from_pretrained(BERT_MODEL_PATH)
config.max_length=4000
config.max_position_embeddings=4000
config.output_hidden_states=True
model = MyBertModel(config)

but still gives me an error mismatch

RuntimeError: Error(s) in loading state_dict for BertModel:
size mismatch for bert.embeddings.position_embeddings.weight: copying a param with shape torch.Size([512, 768]) from checkpoint, the shape in current model is torch.Size([4000, 768]).


texts: How can you build a model that reads out receipts and invoices?

texts:
The objective is to build a model that is capable of identifying information on receipts and invoices that can look completely different.

I've had a discussion with my brother about the right approach. I have attached an example, here the original and below is the important information in boxes:

The green boxes are the must-have information. The one in purple and green indicates that we need either or. The orange information would be a nice-to-have, but not necessarily required.
Some of the boxes have context and inter linkage.

From a data set point of view, we have a sample size of 1,000 receipts and all of them have the necessary information extracted. We could increase the sample size further if that was required.

The approach that I would have chosen:
Treat everyone of the images of the receipts like a game and let the model figure out itself how to arrive at the right conclusion. This will most likely be very computing intensive but I feel like it will be more robust when dealing with new image types.

The approach my brother has suggested:
Basically using the boxes that I've provided and let me model learn from that. The model would then learn to identify the important areas on a receipt or invoice and would go from there. He compared the model to one that would identify license plates.

Edit:
Just to reiterate why I think OCR is only part of the solution but not the solution itself. Here is the Adobe Acrobat OCR result:

Perfect if you ask me. It just doesn't help me figure out what values to use and which to ignore. I don't want to do this manually. I want the model(s) to return for me:

Total amount
Sales Tax & Amount
Creditor (i.e. the company and ideally the tax identifier CHE-xxxxx MWST
Date and ideally time
Payment method

Does this make more sense now? I just don't see how OCR gets me there. It will only be the method to extract the values.


texts:
I am trying to solve poker tournament winner prediction problem. I’ve millions of historical records in this format:

Players ==> Winner
P1,P2,P4,P8 ==> P2
P4,P7,P6 ==> P4
P6,P3,P2,P1 ==> P1

I want to find the best algorithm to predict winner from set of known players.
So far I have tried decision trees, XGboost without much success.

I’ve  done my research and could not find answer anywhere else.My apologies in advance if same problem is answered in different terms on stack-overflow.


texts:
I want to create a spell checker that corrects the spelling mistakes contextually.

For example,

Erroneous sentence: I want to apply for credit cart

Corrected sentence: I want to apply for credit card

Here, the respective spellings of cart and card are correct but the cart is contextually incorrect.

So what methods we can apply for contextual errors like this?


texts: strings as features in decision tree/random forest

texts:
I am doing some problems on an application of decision tree/random forest. I am trying to fit a problem which has numbers as well as strings (such as country name) as features. Now the library, scikit-learn takes only numbers as parameters, but I want to inject the strings as well as they carry a significant amount of knowledge.

How do I handle such a scenario?

I can convert a string to numbers by some mechanism such as hashing in Python. But I would like to know the best practice on how strings are handled in decision tree problems.


texts: Cost tends to infinity when relu activation is used

texts:
I have implemented a neural network with 1 hidden layer using sigmoid activation unit but after watching a video on how relu activation function can be much faster a tried implementing it but the cost function is either Nan or inf in python. I have found that many people have the same problem but I couldn't find any solution on the internet.
I have added '#changed' comment to the lines that I have changed from the previous version in which I used sigmoid activation function.
Related python code snippets:

def relu(arg):  #I have tried both relu and leaky relu
    return 1*(arg<0)*0.0001*arg + (arg>=0)*arg

def reluGrad(arg):
    for i in range(arg.shape[0]):
        for j in range(arg.shape[1]):
            if arg[i][j]>0:
                arg[i][j]=1
            else:
                arg[i][j]=0
    return arg
def softmax(x):
    x = x.transpose()
    e_x = np.exp(x - np.max(x))
    return (e_x / e_x.sum(axis=0)).transpose()

#forward prop:
a1 = np.insert(data,0,np.ones(len(data)),1).astype(np.float64)
    z2 = a1.dot(theta1)
    a2 = relu(z2) #changed
    a2 = np.insert(a2,0,np.ones(len(a2)),1)
    z3 = a2.dot(theta2)
    a3 = softmax(z3) #changed
#compute the cost:
    cost = -(output*(np.log(a3))+(1-output)*(np.log(1-a3))).sum()
    cost = (1/len(data))*cost + (lamb/(2*len(data)))*((np.delete(theta1,0,0)**2).sum() + (np.delete(theta2,0,0)**2).sum())

backProp:
    sigma3 = a3-output
    sigma2 = (sigma3.dot(np.transpose(theta2)))* reluGrad(np.insert(z2,0,np.ones(len(z2)),1)) #changed
    sigma2 = np.delete(sigma2,0,1)
    delta2 = (np.transpose(a2)).dot(sigma3)
    delta1 = (np.transpose(a1)).dot(sigma2)

    grad1 = delta1/len(data) + (lamb/len(data))*np.insert(np.delete(theta1,0,0),0,np.zeros(len(theta1[0])),0)
    grad2 = delta2/len(data) + (lamb/len(data))*np.insert(np.delete(theta2,0,0),0,np.zeros(len(theta2[0])),0)

#update theta
    theta1 = theta1 - alpha*grad1
    theta2 = theta2 - alpha*grad2

What is causing this problem ? How can it be fixed ?


texts: Which direction of the normal vector that defines a Perceptron hyperplane is default?

texts:
This slide here defines a hyperplane by using a normal vector $w$.

And claims that:

  "The hyperplane passes through origin"

Is it a general default parameter? w = [-1,1]?

I see a lot of figures with different directions.

Where $w$ and $b$ play in the opposite direction.


texts:
Using Python, I am trying to predict the future sales count of a product, using historical sales data. I am also trying to predict these counts for various groups of products.

For example, my columns looks like this:

Date Sales_count Department Item Color

8/1/2018, 50, Homegoods, Hats, Red_hat

If I want to build a model that predicts the sales_count for each Department/Item/Color combo using historical data (time), what is the best model to use?

If I do Linear regression on time against sales, how do I account for various categories? Can I group them?

Would I instead use multilinear regression, treating the various categories as independent variables?


texts: Are deep learning models way over the required capacity for their datasets' estimated entropies?

texts:
this question might seem a bit odd. I was doing some self-studies into information theory and decided to do some more formal investigations into deep learning. Please bear with me as I try to explain.
I took a large "training" subset of MNIST as my guinea pig.

1) Converted every image in MNIST into "black-and-white" (pixels values only 0 or 1)

2) Summed over all data images to build a histogram over the pixels - I counted the number of times each pixel gets a 1 value in the dataset

3) Normalized histogram to get an estimate of the "true" probability distribution

4) From this, I got the following probability distribution (shown as a heatmap with matplotlib):

[

5) Now I calculated the entropy and got: $191$ bits

6) According to David MacKay in his Information Theory book, we could interpret a neural network as a noisy channel and consider each neuron as having a 2 bit capacity. Although he does state to use this idea with care. Chapter 40 of his book http://www.inference.org.uk/itila/book.html)

7) So, as a rough estimate (and with care) we could say we would need a neural network of 95 neurons in order to be able to encode the labeling of this MNIST training set (190/2)
8) Now we can get to my question:

Even if this is a very "back-of-the-envelope" calculation, shouldn't a neural network capable of learning the labelling be at least in the ballpark of 95 neurons?
Why do we need, for instance, a neural network with 21840 parameters to get 99% accuracy? (considering the one in PyTorch's example for MNIST: https://github.com/pytorch/examples/blob/master/mnist/main.py)


texts: Strange behavior in current Keras setting (tensorflow-1.7.0 Keras-2.1.5) when using CPU

texts:
After reinstalling Keras and Tensorflow on a Virtual Machine, I have noticed that models fail to train correctly (at the very least those I have checked).
For example, taking this code from the Keras blog as a test case, instead of decreasing loss and final validation accuracy of ~0.8, loss quickly increased to ~7.5 and accuracy stayed at ~0.5.

It should be noted that when I tried it with older versions of Keras and Tensorflow (on the same computer), I did not observe this aberrant behavior.

I have repeated this experiment at three other setups in which the same behavior was observed - a two other computers using a VM, and a third computer, this time a laptop with no suitable GPU.
On the other hand, trying this on two computers with GPUs, the training progressed as expected, with loss decreasing and accuracy increasing to the level advertised in the Keras blog and observed with the previous versions of Keras and Tensorflow while using a CPU.

From the evidence I can speculate that either Keras's or Tensorflow's interaction with the CPU was changed in a manner disrupting the training processes.

Has anyone here faced a similar problem?
What can be done to solve this?


texts: Image feature extraction Python skimage blob_dog

texts:
I am trying to extract features from images using:

def process_image(image_fp):
    image_ = imread(image_fp)
    image_ = resize(image_, (300, 200,3))
    image=equalize_hist(rgb2gray(image_))
    edges = skimage.feature.blob_dog(image)
    return edges.reshape(edges.size).tolist()

where image_fp is an image path.

I am having a problem due to the different sizes of the return. In general, the reshape guarantees the same size in the others algorithms.

Is there a way to get always the same size?

I only see a way: to truncate the lists (the stupid way).


texts: I am using the BERT implementation in https://github.com/google-research/bert for feature extracting and I have noticed a weird behaviour which I was not expecting: if I execute the program twice on the same text, I get different results. I need to know if this is normal and why this happens in order to treat this fact in one or another way. Why is the reason for this? Aren't neural networks deterministic algorithms?


texts: Is there Orange support for MySQL and SQLite?

texts: I would like to be able to directly access both MySQL and SQLite databases as part of an Orange workflow. I see there is support for PostgreSQL and (possibly)


texts: How to combine GridSearchCV with Early Stopping?

texts:
I'm a beginner in machine learning and want to train a CNN (for image recognition) with optimized hyperparameter like dropout rate, learning rate and number of epochs.

The optimal hyperparameter I try to find via GridSearchCV from Scikit-learn.
I have often read that GridSearchCV can be used in combination with early stopping, but I can not find a sample code in which this is demonstrated.

With EarlyStopping I would try to find the optimal number of epochs, but I don't know how I can combine EarlyStopping with GridSearchCV or at least with cross validation.

Can anyone give me a hint on how to do that, it would be a great help?

My current code looks like this:

def create_model(dropout_rate_1=0.0, dropout_rate_2=0.0, learn_rate=0.001):
    model = Sequential()
    model.add(Conv2D(32, kernel_size=(3,3), input_shape=(28,28,1), activation='relu', padding='same')
    model.add(Conv2D(32, kernel_size=(3,3), activation='relu', padding='same')
    model.add(MaxPooling2D(pool_size=(2,2)))
    model.add(Dropout(dropout_rate_1))
    model.add(Dense(128, activation='relu'))
    model.add(Dropout(dropout_rate_2))
    model.add(Dense(10, activation='softmax'))
    optimizer=Adam(lr=learn_rate)
    model.compile(loss='categorical_crossentropy', optimizer=optimizer,
                             metrics=['accuracy'])
    return model

model = KerasClassifier(build_fn=create_model, epochs=50, batch_size=10, verbose=0)
epochs = [30, 40, 50, 60]
dropout_rate_1 = [0.0, 0.2, 0.4, 0.6]
dropout_rate_2 = [0.0, 0.2, 0.4, 0.6]
learn_rate = [0.0001, 0.001, 0.01]
param_grid = dict(dropout_rate_1=dropout_rate_1, dropout_rate_2=dropout_rate_2,
                        learn_rate=learn_rate, epochs=epochs)
grid = GridSearchCV(estimator=model, param_grid=param_grid, n_jobs=-1, cv=5)
grid_result = grid.fit(X, y)


texts: How much of a disadvantage is a small sample size?

texts:
I am examining a petition involving all UK constituencies. In this dataset 2 of the 632 constituencies have not participated in the petition - in terms of data quality how does this affect my examination?

I am examing which parts of the UK tend to vote for left/right ideals.


texts: How to make predictions based on correlations?

texts:
I have correlation values for profit based on three different attributes - attribute1,attribute2,attribute3. Now I want to make predictions for the profit based on these 3 different correlation values.

I wanted suggestions for some efficient algorithms that can help me to make predictions  from these correlation values.

Note: I am new to this so I would appreciate some suggestions if there is some lack of understanding in me.


texts: improve NER model accuracy with spaCy dependency tree

texts:
I have search at lot, was not able to find a solution for my problem...
I am training a NER model, that should detect two types of words: Instructions and Conditions. This is not the standard use-case of NER, as it does not search for specific types of words (e.g. Google == Corporation), but is rather much more depended on the sentence structure.
For example:
If the car crashes, the airbag should go off.

'crashes' should be labeld: "condition"
'go' should be labeld: "instruction"

When training the model, I want to provide for each sentence not only my annotations, but also the dependency tree of the sentence calculated by the 'en_core_web_sm' model. I want my model to not only train based on the given words but also train based on the sentence structure.
My training data currently looks like this, but I want to expand it by also adding the dependency tree of each sentence generated using the 'en_core_web_sm' model:
train_data =
    ("If the car crashes, the airbag should activate", [(11, 17, 'CON'), (38, 46, 'INS')]),
    ...
]

This is my current training loop, using the update function from spaCy, but I am open on trying a different tool:
import random
import datetime as dt
from spacy.util import minibatch, compounding
from spacy.util import decaying

dropout = decaying(0.6, 0.2, 1e-4)

nlp = create_blank_nlp(TRAIN_DATA)
optimizer = nlp.begin_training()
for i in range(80):
    losses = {}
    batches = minibatch(TRAIN_DATA, size=compounding(4.0, 32.0, 1.001))
    for batch in batches:
        texts, annotations = zip(*batch)
        nlp.update(
            texts,               # batch of texts
            annotations,         # batch of annotations
            drop=next(dropout),  # dropout
            losses=losses,
        )
    print(f"Losses at iteration {i} - {dt.datetime.now()} {losses}")

I am curious if and how this is might be possible. It feels like a waste to not use the pretrained model (mind you, the pretrained NER model from spaCy probably will not help me, only the dependency part).
Open to any advice, thank you.


texts:
I have a continuous variable, sampled over a period of a year at irregular intervals. Some days have more than one observation per hour, while other periods have nothing for days. This makes it particularly difficult to detect patterns in the time series, because some months (for instance October) are highly sampled, while others are not.

My question is what would be the best approach to model this time series?

I believe most time series analysis techniques (like ARMA) need a fixed frequency. I could aggregate the data, in order to have a constant sample or choose a sub-set of the data that is very detailed. With both options I would be missing some information from the original dataset, that could unveil distinct patterns.
Instead of decomposing the series in cycles, I could feed the model
with the entire dataset and expect it to pick up the patterns. For
instance, I transformed the hour, weekday and month in categorical
variables and tried a multiple regression with good results (R2=0.71)

I have the idea that machine learning techniques such as ANN can also pick these patterns from uneven time series, but I was wondering if anybody has tried that, and could provide me some advice about the best way of representing time patterns in a Neural network.


texts:
I am working on a convolutional neural network for image classification. The training dataset is too large to be loaded on my computer memory (4gb), on top of that I also need to try some augmentation to balance the classes.

I am using keras. I have looked into many examples but I have not found a solution to my problem. I am thinking about replacing the parameters of the model.fit call by passing a function or generator doing the 'batching'. This function I will design to import from disk and apply augmentation.

I have no idea how to implement that. Any suggestions?


texts: Which statistical test you propose to assess whether there is significant effect of specified categorical variable (s) on a categorical outcome?

texts:
I have a table that I am trying to analyse.
GAME    COMP win
BBall   N   Y
FBall   N   Y
FBall   N   Y
BBall   N   Y
FBall   Y   Y
FBall   N   Y
BBall   Y   Y
FBall   Y   N
BBall   N   Y
FBall   N   Y

John plays football and baseball under competitive and non-competitive conditions.
He wants to know which he is better at.

What is a good statistical test to determine which game he is better at? I think Chi-Square test. Does that make sense?
Are there some useful visual charts that can be used to show this?
Also, an informative chart to show wins vary by game or competitiveness?


texts:
I am getting the following error while using xgboost.cv() (scikit-learn interface). I am working on a regression problem. Below is the code and trace. No idea why it is giving this error.

from hyperopt import hp
from hyperopt import fmin, tpe, hp, STATUS_OK, Trials

space = {
             'n_estimators' : hp.quniform('n_estimators', 100, 1201, 100),
             'eta' : hp.loguniform('learning_rate', np.log(0.01), np.log(0.2)),
             'max_depth' : hp.choice('max_depth', np.arange(1, 14, dtype=int)),
             'min_child_weight' : hp.quniform('min_child_weight', 1, 6, 1),
             'subsample' : hp.quniform('subsample', 0.5, 1, 0.05),
             'gamma' : hp.quniform('gamma', 0.5, 1, 0.05),
             'colsample_bytree' : hp.quniform('colsample_bytree', 0.5, 1, 0.05),
             'eval_metric': ['rmse'],
             'objective': ['reg:linear'],
             'n_jobs' : [-1]
             }
dtrain = xgb.DMatrix(X_train, label=y_train)
cv_results = xgb.cv(space,dtrain,num_boost_round = 1000, nfold= 5, obj = ['reg:squarederror'],
                        stratified = False, early_stopping_rounds = 100, metrics="rmse", seed = 44)

cv_results

The following is the part of the trace:

---------------------------------------------------------------------------
XGBoostError                              Traceback (most recent call last)
<ipython-input-19-b45243537e3b> in <module>
      1 dtrain = xgb.DMatrix(X_train, label=y_train)
      2 cv_results = xgb.cv(space,dtrain,num_boost_round = 1000, nfold= 5, obj = ['reg:squarederror'],
----> 3                         stratified = False, early_stopping_rounds = 100, metrics="rmse", seed = 44)
      4
      5 cv_results

...
...
...
XGBoostError:src/objective/objective.cc:23: Unknown objective function ['reg:linear']\n\nStack trace returned 7 entries:\n[bt]

Any suggestion as to how to avoid this would be helpful.
Thanks


texts: Understanding difference between benefits of having multi-head attention and of the process of learning Q,K,V embeddings in a single head

texts:
From reading different papers and blogposts I got an understanding that learning embeddings with a single attention head and with multiple heads serves essentially the same purpose.
In this blogpost it is implied that by learning Q,K,V embeddings it is possible to learn subject-to-verb relation but not the other possible relationships. For those others relationships (e.g. possessive-to-noun) we need to use multi-head attention mechanism.
Questions are:

what makes it possible to learn subject-to-verb relationships during embedding learning and prevents it from learning other types of relationships? Does it mean some relationships are more "obvious" to that algorithm and if yes which ones and why?
if having multiple heads just adds more diversity and enables us to learn more relationships why isn't there a chance to learn possessive-to-noun relationship with a single head only? Is it random of we can control what kind of relations we are learning?


texts: Machine learning algorithm for a dynamic set of attributes

texts:
I'm new to machine learning programming and am working on an application which will calculate weights of the attributes to calculate the ETAT (Estimated Turn Around Time) of some tasks in workflow. The set of attributes is not fixed.

For example: ETAT will depend on-

Language
No of properties
Experience level of resources and many
more.

For each workflow entity there can by a set of these attributes, i.e. a single workflow may be like:

Language - English
No. of properties - 1200
Experience level of
resources - Expert

and

Language - Russian
No. of properties - 21000
Experience level of resources - Basic

Now for the above set of attributes, how can I design my machine learning algorithm? Any pointers will be highly useful.


texts: Help in KERAS, a model.fit method on a array 50x50 slows the system, how can I make it faster?

texts:
i m a new member and i am a student. i have a problem. how can I make this training faster my neural network ? when I call the fit method on a 100 x 100 matrix goes very slow
my model it's a sequential
def build_model(env, **opt):
loss = opt.get('loss', 'mse')
a = opt.get('alpha', 0.24)
model = Sequential()
esize = env.maze.size
model.add(Dense(esize, input_shape=(esize,)))
model.add(LeakyReLU(alpha=a))
model.add(Dense(esize))
model.add(LeakyReLU(alpha=a))
model.add(Dense(num_actions))
model.compile(optimizer='adam', loss='mse')
return model

this is my Experience Class
def __init__(self, model, max_memory=100, discount=0.97):
    self.model = model
    self.max_memory = max_memory
    self.discount = discount
    self.memory = list()
    self.num_actions = model.output_shape[-1]

def remember(self, episode):
    # episode = [env_state, action, reward, next_env_state, game_over]
    # memory[i] = episode
    # env_state == flattened 1d maze cells info, including agent cell (see method: observe)
    self.memory.append(episode)
    if len(self.memory) > self.max_memory:
        del self.memory[0]

def predict(self, env_state):
    return self.model.predict(env_state)[0]

def get_data(self, data_size=10):
    env_size = self.memory[0][0].shape[1]   # env_state 1d size (1st element of episode)
    mem_size = len(self.memory)
    data_size = min(mem_size, data_size)
    inputs = np.zeros((data_size, env_size))
    targets = np.zeros((data_size, self.num_actions))
    for i, j in enumerate(np.random.choice(range(mem_size), data_size, replace=False)):
        env_state, action, reward, next_env_state, game_over = self.memory[j]
        inputs[i] = env_state
        # There should be no target values for actions not taken.
        # Thou shalt not correct actions not taken #deep (quote by Eder Santana)
        targets[i] = self.predict(env_state)
        # Q_sa = derived policy = max quality env/action = max_a' Q(s', a')
        Q_sa = np.max(self.predict(next_env_state))
        if game_over:
            targets[i, action] = reward
        else:
            # reward + gamma * max_a' Q(s', a')
            targets[i, action] = reward + self.discount * Q_sa
    return inputs, targets

when i call model.fit , the system speed slows
 h = self.model.fit(
        inputs,
        targets,
        epochs = epochs,
        batch_size=16,
        verbose=0,
    )

the matrix that i step to it is
n = 50

def build(n):

mat=np.ones(n*n)

return mat.reshape((n,n))

should I modify the template by adding more Danse layers? I don’t know how to improve the performance because I would like to call it on a 100x100 matrix


texts: Addressing mechanisms in neural turing machine

texts: In a Neural Turing Machine,why wasn't an absolute random access mechanism used?We are reading and writing based on the weighting emitted by the read/write head and this weighting is being generated by addressing mechanisms:content based addressing and location based addressing.Can anyone explain the need for generating these weightings with these mechanisms.


texts: Should the cost function be zero using TensorFlow's sigmoid_cross_entropy_with_logits?

texts:
I'm building a CNN to make a binary classification (1 or zero). For this, I'm using the cost function sigmoid_cross_entropy_with_logits.

But for some reason, the cost using this function is never equal to zero even if the prediction is equal to the correct valuel.

I tried plotting the output using the formula on TensorFlow's website:
https://www.tensorflow.org/api_docs/python/tf/nn/sigmoid_cross_entropy_with_logits

This formula:

max(x, 0) - x * z + log(1 + exp(-abs(x)))

And by making this plot, I realized that it really isn't zero when the outputs are equal. For example, if z = 0 and x = 0, the result of this function is ~0.693.

This isn't really making sense to me. Can someone shed some light on why it isn't zero when the prediction is correct?


texts: Concatenating branches (of different dimensions) of minigooglenet on Pytorch

texts:
Is there any way I can concatenate branches of different dimensions of a small inception(googlenet)?
For example, at a 32 x 32 x 3 image (torch.Size([1, 3, 32, 32])), it will pass through an inception module with the following branches:

a convolution with (32 channels, 1 x 1 filters)
another convolution with (32 channels, 3 x3 filters)
merge (concatenate along the channel dimension)

[![enter image description here][1]][1]
[1]: https://i.stack.imgur.com/TILMQ.png
The issue however is that the torch sizes as a result of the two convolutions are different.

(32 channels, 1 x 1 filters) -> [1, 32, 30, 30]
(32 channels, 3 x3 filters) -> [1, 32, 28, 28]

How can I concatenate the two branches? Should I add padding? I tried ZeroPad2d. It’s not working.
For your reference:
class Inception(nn.Module):
     def __init__(self, in_channel, ch1, ch3):
          super(Inception, self).__init__()
          self.branch1 = nn.Sequential(
               ConvBlock(in_channel, ch1, kernel_size = 1,stride=1, padding=0)
          )

          self.branch2 = nn.Sequential(
               ConvBlock(in_channel, ch3, kernel_size = 3,stride=1, padding=0)
         )

     def forward(self, x):
          branch1 = self.branch1(x)
          branch2 = self.branch2(x)
          return torch.cat([branch1, branch2], 1) *(error here)*


texts: How to recreate T-SNE dimensions deterministically?

texts: So I have a set of 3000 features from which I would like to generate clusters. I passed my features through the T-SNE algorithm to reduce dimensionality to 2 features, and clusters are really visible in this reduced space. How would I go about figuring out a deterministic way to create this same T-SNE output to use for clustering?


texts:
I want to create a heat map to visualize some production data, but without geolocation. I am finishing some experiments in a greenhouse, divided in different sectors. The idea is to make a heat map to watch in which areas we are harvesting more fruits, or less, in each season.

Which tool, software or language do you think is better for this purpose? I was looking for information, but the main tools are for geolocation.


texts: Does importance of SVM parameters vary for subsample of data?

texts:
I'm training a support vector machine classifier (SVM) on 100.000 observations. I'd like to try different parameter combinations (including kernel types) using cross-validation. However, it is computationally expensive to optimize over these different parameter values. Therefore, I'd like to test them on a subset of 5.000 observations on a validation set and then based on those results choose the parameters for the final model that is trained on 100.000 observations.

My question is then whether there is something intrinsic in these parameters that will make this approach problematic. More specifically, is there a parameter, where training on the full data set will require a different value for optimal performance?

I'm using the following kernels with their parameters in parentheses:

rbf (gamma, C)
poly (degree, C)
sigmoid (gamma, C)
linear (gamma).


texts: If i use use BERT embeddings for if cosine(sent1,sent2) > 0.9, then is it fair to assume s1 and s2 are similar

texts: According to BERT author Jacob Devlin: I'm not sure what these vectors are, since BERT does not generate meaningful sentence vectors. It seems that this is doing average pooling over the word tokens to get a sentence vector, but we never suggested that this will generate meaningful sentence representations. And even if they are decent representations when fed into a DNN trained for a downstream task, it doesn't mean that they will be meaningful in terms of cosine distance. (Since cosine distance is a linear space where all dimensions are weighted equally).


texts:
I created a Random Forest regressor model and calculated my own error. I also want to calculate MAE, MSE and RMSE to compare my results to similar use cases. I am confused by the results as the values are far lesser.
The random forest regressor function I used is the default one. So I am just curious if there is anyone who can interpret my MAE, MSE, RMSE? Or did I do something wrong?
# Random Forest Regressor trainieren
random_state = 42

rf.fit(X_train, y_train)

These are the results of my own error:
Testset size: 19688
mean error(AVG): 0.10146151388130142
median error (Median): 0.061079373768676316

These are the results that look confusing to me:
# predict the testset
y_pred = rf.predict(X_test)

# Random Forest Basline Model evaluation
print('Mean Absolute Error:', metrics.mean_absolute_error(y_test, y_pred))
print('Mean Squared Error:', metrics.mean_squared_error(y_test, y_pred))
print('Root Mean Squared Error:', np.sqrt(metrics.mean_squared_error(y_test, y_pred)))

#Mean Absolute Error: 0.027392195979941525
#Mean Squared Error: 0.00142917555312868
#Root Mean Squared Error: 0.03780443827288907


texts: How can I tell whether my Random-Forest model is overfitting?

texts:
I was trying to generate predictions for Iris species using the UCI Machine Learning Iris dataset. I used a RandomForestClassifier with GridSearchCV and calculated the mean absolute error. However, upon generating predictions with the testing set it gave me a suspicious MAE of 0.000000, and a score of 1.0. Is it likely that the model is overfit? If so, why did this happen, and how do I prevent this?

iris = pd.read_csv('/iris/Iris.csv')

le = LabelEncoder()
i2 = iris.copy()
labelled_iris_df = pd.DataFrame(le.fit_transform(i2.Species)).rename(columns={0:'Species_Encoded'})

i3 = i2.drop('Species', axis=1)
i3 = pd.concat([i3, labelled_iris_df], axis=1) #Encoded dataset

y = i3.Species_Encoded
X = i3.drop('Species_Encoded', axis=1)

X_train, X_test, y_train, y_test = train_test_split(X, y, train_size=0.75, random_state=42)

params = {
    'n_estimators':[50,100,150,200],
    'max_depth':[3,4,5,6]
}

rfc = RandomForestClassifier(random_state=42)
gc = GridSearchCV(rfc, params, cv=3).fit(X,y)
print (gc.best_params_) #n_estimators: 50, max_depth:4

model = RandomForestClassifier(n_estimators=50, max_depth=4, random_state=42)
model.fit(X_train,y_train)
preds = model.predict(X_test)
mae = mean_absolute_error(y_test, preds)
sc = model.score(X_test, y_test)
print("mae: %f \t\t score: %f" % (mae, sc)) #Prints mae: 0.000000   score: 1.0

I'm a beginner to Machine Learning so please feel free to comment on bad sections of this code and how I can improve them.


texts:
I am going through the "Text classification with TensorFlow Hub" tutorial.
In this tutorial, a total of 50,000 IMDb reviews are split into 25,000 reviews for training and 25,000 reviews for testing.

I am surprised by this way of splitting the data, since I learned in Andrew Ng's course that for fairly small datasets (<10,000 examples) the "old-fashioned" rule of thumb was to consider 60% or 70% of the data as training examples and the remainder as dev/test examples.

Is there a reason behind this 50:50 split?

Is it common practice when working with text?
Does it have anything to do with using a "pre-trained" TensorFlow Hub layer?


texts: Accessing the intermediate layers of a tensorflow cnn model (Unity)?

texts:
I'm new to scripting in Unity and I am doing a project that takes in video from my laptop's camera, then displays a heatmap in AR using the information in an intermediate layer from a CNN model.

I've checked out tutorials online but I haven't seen anyone discuss accessing intermediate layers from a model. I've also read the tensorflowNET, tensorflowSharp and ml-agents documentations but I haven't seen this subject discussed.

Essentially I would like to reproduce this but in Unity/C# (Example is in python).

    intermediate_layer_model = Model(inputs=model.input,
                                     outputs=model.get_layer(layer_name).output)

    layer_output = intermediate_layer_model.predict(Image)

Is this possible and can anyone point me in the right direction?

Cheers.


texts: What is the best way to organize the datasets for my task?

texts:
I am working on a task — geolocation estimation of Twitter users by using tweets only. I collected tweets (and users) from more than 6000 people in Twitter. Each user is associated with a city.

In the dataset, number of samples (or users) for each city depends on the city size. (i.e. If the city A is more populous than the city B, the city A has more users in the dataset.) This seems fair, but it creates an unbalanced dataset.

Right now, I am planing to collect another dataset; a dataset that is more balanced (i.e. there will be the almost the same amount of users for each city although there will still be more users in big cities). Doing this makes sense or should I continue with the unbalanced dataset? What approach would it be good for that task?


texts: From most frequent words how to extract technical skill words

texts:
I've scrape 30 job description web and stored them into a list called job_desc where each item is a job description.

# each item is a list of tokenized job_description
tok = [nltk.word_tokenize(job.lower()) for job in job_desc]

# ignore stop words, bullets, etc. And put it into one list
from nltk.corpus import stopwords
stop = stopwords.words('english')

def clean_token(what_to_clean):
    cleaned_tok = []
    for lists in what_to_clean:
        for item in lists:
            if len(item)>2 and (item not in stop):
                cleaned_tok.append(item)
    return cleaned_tok

After cleaning job description I've found most frequent words using:

freq = nltk.FreqDist(clean_token(tok))
most_freq_words = freq.most_common(100)

Which outputs:

 [('data', 211),
 ('experience', 78),
 ('learning', 70),
 ('business', 65),
 ('team', 53),
 ('science', 51),
 ('machine', 48),.....

From here I only want to extract words like machine, python, C+, technical skills. How can I go about it?

Also you can see there is word "machine" showing up 48 times and I am not sure whether it is talking about machine learning how can I go about this, I know if I want to make predictions I could've used CountVectorizer and n-grams.


texts: Should a beginner in ML use libraries like TF and Keras?

texts: As a beginner in ML field, should I in beginning implement various algorithms from scratch (using only NumPy for example) and then start using ML libraries like TensorFlow, Keras or vice-versa? For example, I implemented multilayer feed-forward neural network in Python using only NumPy but it took me few weeks to understand NNs at the level that I can implement them from scratch, would it be better that I spent that time to cover theory less thoroughly and use libraries to make more advanced models easily?


texts: Unable to use pip package obtained from building Tensorflow 2.3 from source

texts:
I've managed to build Tensorflow 2.3 from source, following these instructions:
https://towardsdatascience.com/how-to-compile-tensorflow-2-3-with-cuda-11-1-8cbecffcb8d3
But, when I install obtained pip package in new conda environment, and import tensorflow, I get following error: Could not load dynamic library 'libcudart.so.11.1'; dlerror: libcudart.so.11.1: cannot open shared object file: No such file or directory
I've managed to use GPU support with CUDA 11.1 for Tensorflow 2.5 nightly, without creating soft links between libs (I get Successfully opened dynamic library libcudart.so.11.0 message).
Any help appreciated.


texts: Is there a mistake in Lecture 5 of Stanford CS234 available on youtube?

texts:
https://www.youtube.com/watch?v=buptHUzDKcE&list=PLoROMvodv4rOSOPzutgyCTapiGlY2Nd8u&index=5
At 53:45 Professor starts to describe temporal difference for linear value function approximation. At 56:20 on slide one can see how weights are updated. Is equation for $ \Delta w $ correct?
In my opinion thing in brackets should be multiplied by $ X(s) - \gamma X(s') $ instead of $ X(s) $ because $ \frac {\partial ( \gamma X(s')^T w )} {\partial w} $ is not zero.
Am i right?


texts: Is the external memory of Differtiable Neural Computer limited?

texts: Is there a CONSTANT limit in DNC external memory usage?  Like a human brain, 7 chunks of information? Or is it some sort of a hyperparameter? How much data it can associate using that Short-Term memory?


texts: How do GD, Batch GD, SGD, and Mini-Batch SGD differ?

texts:
How do these four types of gradient descent functions differ from each other?

GD
Batch GD
SGD
Mini-Batch SGD


texts: I was reading the Keras documentation on shared layers, and used the example of a model built to classify whether 2 tweets were from the same user or not.  Is there any dataset that I could use to do this?


texts: Using Random Forest Regression correctly

texts:
I have run Random forest regression with python and I have the fear that I haven't done it correct.

I have original image that has 3 different bands (each pixels has 3 values) and I want to try to see if I can predict the first value using the two others.

For that I used Random Forest regression, I have created train and test and fit the model:

rf.fit(X_train,y_train)
rf_pred=rf.predict(X_test)

then after I checked the prediction on the test and saw it was good, I wanted to use the same model in order to predict all the values of all the pixels in the image, so I did that:

pred_all=rf.predict(data)

*data includes all the pixels of the image.

My question : is this the right way to do this? can I just predict all the  pixel values just by using the rf.predict after fit it with the train and test sets? Or am I missing here some step that should be taken?


texts: How would you categorise email subjects to find similar emails?

texts:
I have a list of email subjects like

<XYZ> commented on <ABC>
Weekly review for <Company>
Your account is ready

And I want to find patterns in them so I can group them.

Is there a well known algorithm I can use?


texts:
TLDR: Please help me understand the graph representation of the network in the image below.

Hi, this is pretty stupid, but I'm just have trouble visualising what I'm actually doing with this neural network.

I've read about neural networks and multilayer perceptrons for some time and I'm just getting started with actually using them. I started with a super simple example, just to get warmed up but now I've confused myself.

I artificially generated some data and used nntools in matlab to attempt to "predict" the results. I built a neural network with the following parameters:

feed forward backprop network.
Gradient Descent training algorithm.
Gradient Descent learniing algorithm.
Performance/loss function of mean squared error.
two layers:

first with three neurons and Tansig activation function.
the second with one neuron and linear activation.

I end up with something looking like this:

However, I don't know what this actually represents, I'm all sorts of confused right now. Could someone please explain/upload an image/draw some ascii to represent the neurons and edges in the above network? It would really help clear my head.

Currently I think it's like this:

           T  L
           o
          / \
         /   \
   IN > o--o--o--o > OUT
         \   /
          \ /
           o

With linear activations in columns L and Tanh activations in columns T. Is that right? Doesn't make sense to me.


texts: How does keras handle convolution in case the size of the output is less than the possible number of extracted features using convolution?

texts:
All is in the question:
Consider the case we specify an output size of 64 but with current convolution layer parameters we can get 128 features.
How is this handled in general and using keras in particular?


texts: How to generate list with out for loop in python

texts:
data frame

S/N Type    Number Capacity
1   Bike      2      5
2   Tempo     1      30
3   Truck-1   1      60
4   Truck-2   1      90

I would like to generate capacitylist = [5,5,30,60,90]

Is it possible to do it with out for and using map function in python. Thanks Alot.


texts: How is the Beeswarm plot better than a histogram?

texts: I have read about Beeswarm and Histogram plots. But I did not understand the difference. In a Histogram, if the bins number is increased then its intrepretation is changed. So how can a beeswarm plot help to understand the data?