texts: Why is my training score below my testing score?

texts:
I'm learning data science, and currently practicing with the Titanic Dataset. I'm doing a simple logistic regression using scikit-learn, and plotting the learning curves of that model with Matplotlib:
model = LogisticRegression()

X = data[["Pclass","SibSp","Parch","Fare"]]
y = data["Survived"]

lc = learning_curve(model, X, y, train_sizes =[50,100,150,200,250,300,350,400,500,550,600,700])

train_scores = [score.mean() for score in lc[1]]
test_scores = [score.mean() for score in lc[2]]
plt.plot(lc[0],train_scores, color = "r")
plt.plot(lc[0],test_scores);

Learning Curves
I don't understand why my training score is below my testing score at some point.
To be fair, the 2 curves are very close, and since learning_curve is using k-folds and these curves are actually a mean, you can plot a standard deviation around them (see below).
Learning curves with standard deviation
We see that statistically they are basically the same curve. But:

Again, we're using k-folds, so I have the feeling that it should "remove" that kind of randomness. I tried increasing the number of folds (up to 30), and I get the same result.
I shuffled my dataframe several times and I still have the same result.

I know a similar question has been asked, but it didn't answer mine. Also, here I'm using a popular library on a well-known dataset in a very simple way. Any idea how to explain that? Thanks!


texts: Is there any PageRank-like method on weighted graph?

texts:
I found that PageRank algorithm depend heavily on the existence of edges, but the weights of edges give small effect.
For example, PageRank on triangle graph(A -> B,  B -> C,  C -> A) is always (1/3, 1/3, 1/3) even if one of the weight is 0.0000001.
Is there any PageRank-like method work well on the weighted graph?

Thanks in advance!


texts: Display Images (url) Inside Pandas Dataframe

texts:
I would like to display images (mostly jpg and png formats) directly from their url link inside a pandas dataframe. Imagine I already have the following dataframe:

id image_url
1  http://www.nstravel.ro/wp-content/uploads/2015/02/germany-profile_6023_600x450.jpg
2  https://www.nh-hotels.com/multimedia/images/cityscape-of-berlin_retoque_600x458-tcm41-132903-32.jpg
3  https://www.scandichotels.com/imagevault/publishedmedia/8q33xlshjl4o9d5ftp3r/germany-berlin-berliner-dom-cathedral-in-the-eveni.jpg

I would like to have the actual image displayed in another column (properly resized to fit).

I am aware that I could do it differently, e.g. not using pandas and go with grids using matplotlib, but for ease of illustration and presentation I would prefer to show it within a table (dataframe).

I have found these solutions: solution1, solution2. These suggested solutions while seem very relevant but did not work for me. Even I tried with images from my local disk, but I get this using the solution2:

Python: 3.6.5, Pandas: 0.23.0, Jupyter: 4.4.0

Thanks!


texts: Power BI : How to concactenate a variable number of columns?

texts:
I am importing data from Jira in PowerBi directly from the web.
The data contains an unknown number of columns with similar name (for example Comment1, Comment2, Comment3, etc. depending on the maximum number of comments in the issues selected).
I don't know how many of these columns will exist and this is variable.
Is there a way I can concatenate all columns with the same name ? (so far, I found this general solution)
Ideally I would like to concatenate only when there is a value.


texts: Growth function of the class of all circles in the plane

texts:
I know that the VC dimension for this problem is 3. My concern is about the growth function. The following bound is obtained using the VC dimension:
$$m_{\mathcal{H}}(n)\le \sum_{k=0}^3\,{n\choose k}$$
When I simulate this hypothesis set up to $n= 10$, the maximum number of dichotomies I obtain is exactly $\sum_{k=0}^{3}\,{n\choose k}$. Is it by coincidence or is there a way to prove this?


texts: How to generate a random sample and distribute values based in an probability distribution?

texts:
I want to generate a random sample based on this probability distribution:

The line is the KDE of the histogram.
My random sample will have n values, the value is a number of points. Each of the n values generates an amount of points p that must be distributed among the population. So I must distribute the total of n * p points. The distribution of points must follow the probability distribution above.
How should I generate a random sample that follow this probability distribution?
Probably this is a usual problem, so I welcome any help to better formulate my question.


texts: Imbalanced Dataset (Transformers): How to Decide on Class Weights?

texts:
I'm using SimpleTranformers to train and evaluate a model.
Since the dataset I am using is severely imbalanced, it is recommended that I assign weights to each label. An example of assigning weights for SimpleTranformers is given here.
My question, however, is: How exactly do I choose what's the appropriate weight for each class? Is there a specific methodology, e.g., a formula that uses the ratio of the labels?
Follow-up question: Are the weights used for the same dataset "universal"? I.e., if I use a totally different model, can I use the same weights or should I assign different weights depending on the model.
p.s.1. If it makes any difference, I'm using roBERTa.
p.s.2. There is a similar question here, however, I believe that my question is not a duplicate because a) the attached question is about Keras where my question is about Transformers, and b) I'm also asking about general recommendations of how weight values are decided where the attached question is not.


texts: How to approach an extremely unbalanced time series dataset

texts:
I need to classify a relatively small time series dataset.

Training set dimensions are 5087 rows (to classify) by 3197 columns (time samples) which are (or should be as far as I understood) the features of the model. I don't know yet if every sample is important and I will think about downsample/filtering/fourier transform later.

Unfortunately dataset is extremely unbalanced: only 37 (0.7%) out of 5087 rows are "Positive". How would you approach this? I will have to use Scikit-learn library.

Since this is my first approach with Scikit-learn I wanted to try a very simple classifier, with few hyperparameters,and build up from there.

First, choosing the classifier: logistic regression because is the easiest I can think of an this is just a test.
Second, choosing regularization parameter via tuning grid
Third, choosing the splitting cross validation strategy: I wanted to use stratified bootstrap but unfortunately it is not provided by the library so I opted for Stratified shuffle split
Fourth, choosing the metric: cohen's kappa because the dataset is so unbalanced the make accuracy too much biased

Script:

classifier = LogisticRegression(tol=1e-4, max_iter=500, random_state=1)
param_grid = {'C': list(range(3))}
splitter = StratifiedShuffleSplit(n_splits=5, random_state=1)
grid_searcher = GridSearchCV(classifier, param_grid, cv=splitter, scoring=make_scorer(cohen_kappa_score))
model = grid_searcher.fit(train_x, train_y)

First is "cv=splitter" legit? Second, what do you think of this approach? Obviously with such a trivial classifier the model predicted all Negative and I also got some warnings:

  FitFailedWarning: Estimator fit failed. The score on this train-test
  partition for these parameters will be set to nan ZeroDivisionError:
  float division by zero


texts: How to handle multi-label feature for binary classification problem?

texts:
I have dataset like :

   profile     category  target
0        1      [5, 10]       1
1        2          [1]       0
2        3   [23, 5000]       1
3        4  [700, 4500]       0

How to handle category feature, this table may have others additional features too. One hot encoding lead to consume too much space.because number of rows is around 10 million. Any suggestion would be helpful.


texts:
Is it possible to create an LSTM in PyTorch where the time steps are varying? For example, heights where measurements are taken at various times. The data might look like this:

Person id
Inches tall
Date

1
12
2020-01-01

1
15
2020-03-09

1
32
2020-04-01

1
40
2021-01-01

2
38
2020-05-20

...

Given a history of 4 heights and dates, I'd like to get a prediction of a height at a particular date:
lstm(torch.tensor[(21, '2020-01-02'),
                  (23, '2020-02-02'),
                  (29, '2020-04-12'),
                  (41, '2020-12-02')],
     '2022-01-01')

I passed in '2022-01-01' after the 4 tuples above because I want to get the prediction for my height on this date.
Is this possible? How can I do this?


texts: Clustering with k-means for text classification based on similarity

texts:
I have a column that contains all texts that I would like to cluster in order to find some patterns/similarity among each other.

Text
Word2vec is a two-layer neural net that processes text by “vectorizing” words.

Its input is a text corpus and its output is a set of vectors: feature vectors that represent words in that corpus.

While Word2vec is not a deep neural network, it turns text into a numerical form that deep neural networks can understand.

Documentation and examples for common text utilities to control alignment, wrapping, weight, and more.

NounEdit. text (countable and uncountable, plural texts). A writing consisting of multiple glyphs, characters, symbols or sentences.

Fish are gill-bearing aquatic craniate animals that lack limbs with digits.

I tried to determine the optimal number of clusters with elbow method, but unfortunately with no success.
So I remove the stopwords and vectorised texts in the column:

import re
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.cluster import KMeans
from nltk.tokenize import word_tokenize
from nltk.stem import WordNetLemmatizer
import nltk
from nltk.corpus import stopwords
from sklearn.pipeline import Pipeline

stop_words = stopwords.words('english')

def preprocessing(line):
    line = re.sub(r"[^a-zA-Z]", " ", line.lower())
    words = word_tokenize(line)
    words_lemmed = [WordNetLemmatizer().lemmatize(w) for w in words if w not in stop_words]
    return words_lemmed

  vect =TfidfVectorizer(tokenizer=preprocessing)
    vect_text=vect.fit_transform(df['Text'])
    kmeans =KMeans(n_clusters=3).fit(vect_text)

       n_clusters=3, n_init=10, n_jobs=1, precompute_distances='auto',

Is it right to cluster the texts above based on their similarity using this approach?


texts:
I would like to find a way/algorithm to group people into, say, four groups by their answer similarity to yes/no questions.
So, each pair of people in one group would have given the same answers for a big part of the questions – (mostly) bigger than compared to people from the other three groups.
Would appreciate any pointers to the right direction.


texts: a data point - too little to spend resources during data preparation?

texts:
I am learning analytics online and have some quick questions.

Usually when we do analysis, why is that we usually ignore the items/data points that are less frequent?

Let's say for ex: we have is drug frequency data and no of patients who consumed that drug in a hospital. As an example, the data looks like as shown below but in real-time, I might even have millions of records

From the above screenshot, we can know that whatever analysis and insights we come up with the above data (including few more columns of data which aren't shown here), we will definitely not consider Drug D.

Meaning we can't base our conclusion/insights that we derive from our data based on Drug D because only 2 out of 5000 patients have it which is less than 0.05% of our data.

Through it seems to intuitively make sense because 0.05% is very less to have any impact on output.

Now my question is what about Drug G. It occurs 1.14% times in our data?

How do I know it's okay to believe that Drug D - 0.05% is very less to have any impact on output and can be ignored whereas Drug G - 1.14% has to be retained?

I am not sure whether my English skills helped you understand what I am trying to convey.

EDIT - UPDATED (Apologies if my question wasn't clear earlier)

What I am trying to do is (not an ML task but a Data Preparation task), manually map the drug names to the terms available in the dictionary (Data Preparation task). As you can see in the screenshot, Drug A is mapped to ABCDE A. Similarly, I have to manually map for all 50K drugs. However, my question is given below

a) I can't spend resources (money/people) to manually (as it cannot be automated) go through all the 50K drugs and map it to dict terms because no one is interested to do this job. Whoever is interested, is not willing and it would be impossible to do all the 50K drugs and it would incur so much money to pay them. So, I have to make sure that manual reviewers focus on important (high frequent) terms first and it's even okay to ignore DRUG D or DRUG G because they contribute very little value to the data (considering the full dataset of million records)? Question is mainly on decision making based on systematic approach/mathematical approach rather than my judgement/visual inspection/subjective..

b) Hence now, I am trying to know whether there is any objective/systematic/mathematical approach that can tell me, we can ignore all drugs below a certain N% etc... Because I can't just say that through visual inspection I feel Drug G and Drug D can be ignored. If you are gonna suggest me Statistical significance test, can you please guide me on how can I set this as a problem? Because I usually see, it is used in hypothesis testing. Can I kindly request you to guide me on this?


texts: Is it possible to cluster data according to a target?

texts:
I was wondering if there exists techniques to cluster data according to a target. For example, suppose we want to find groups of customers likely to churn:

Target is churn.
We want to find clusters exhibiting the same behaviour according to the fact that they are likely to churn (or not). Therefore, variables not explaining churn behaviour should not influence how clusters are built.

I have done this analysis the following way:

Predict target (e.g. using a Random Forest) and retrieve "most important features" (from feature importance analysis).
Cluster samples with selected features (e.g. using k-means).

However, I am afraid the clustering technique used in the 2nd step might not catch behaviours found in the 1st step which might explain churn (suppose there is a complex interaction in some trees in the RF, this interaction might not be cought in the k-means algorithm).

I was thinking of another way of doing this by using a neural network:

Predict target using a neural network with several layers, and for each sample retrieve activations from a given layer.
Cluster samples with their activations.

If the performance of the neural network is good and if the layer from which activations are retrieved is carefully chosen (not too close to the input or the output layer), I suppose the clusters could show customers displaying the same behaviour explaining the target.

I did not find any articles having this approach. Did anyone deal with the same issue or have other ideas?


texts: How to make a keras prediction from a trained model

texts:
I have a trained model for predicting the property of an image and have some images where the property is unknown. This is what I was training my model to predict for. I'm confused about the dimensions needed for the data to be predicted. When using the model.predict_classes(), I get an error saying that it expects a 4D array. After reshaping it, it says that it expects an array with 3 dimensions.

I think I am misunderstanding the error in some way.


texts: How to a object type data which contains a string of unique id for feeding in my model?

texts:
I am having a dataset in which data type of unique id of a user is in object form. I need to convert it into Int for feeding this data into my model.

here is first rows of my dataset.

event_id             event_time        user_ids       tag_from      tag_to
gse:161a4c6b24541287  2018-02-17 17:18:59   user_out_0  0

I am trying to convert my "event_id" column into a data type which can be used with ml models. here is the output of data.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 1422907 entries, 0 to 1422906
Data columns (total 5 columns):
event_id      1422907 non-null object
event_time    1422907 non-null datetime64[ns]
user_ids      1422907 non-null object
tag_from      1422907 non-null uint8
tag_to        1422907 non-null uint8
dtypes: datetime64[ns](1), object(2), uint8(2)
memory usage: 35.3+ MB}

Thanks in advance.


texts: What predictive model to use to impute Gender?

texts:
My data looks like this:

birth_date has 634,990 missing values
gender has 328,849 missing values

Both of these are a substantial amounts since I have 900k entries, so I can't discard empty rows. For birth_date someone recommended using Multivariate imputation by Chained equation (MICE). I don't know what predictive model I should use for gender. Of the non-missing data, there are 5x more males than females.

Can someone tell me what would be best practice here? What would be the best way to fill in the missing values for gender ?

I'm using the data to predict bike-ride duration and final destination (I know they're shown on the table above)


texts: Why does a machine learning algorithm need a bias?

texts:
The first line of section 2.7.3 in Mitchell's Machine Learning is:

"A Learner that makes no prior assumptions regarding the identity of the target concept has no rational basis for classifying any unseen instances."

Why is it that a machine learning algorithm needs a bias? Can someone please help me understand this, with examples perhaps?


texts: I have a data set of optimal values after simulations, How can I find if this dataset follows a specific pattern or any relation exists?

texts:
In the simulation I am conducting, I have a set of triangles and I select the optimal triangle based on my metric. After every simulation, I obtain an optimal triangle and I note down the lengths of its three sides. So I conducted this simulation a large number of times and noted the lengths of the three sides of the optimal triangle. Now I want to see if there is any pattern which these three sides follow or if there is any relationship among these three sides, or even between two sides.
For example, the names of the three sides are D1, D2, and DX, a relationship can be something like , D1=D2=Dx or D2=2 times Dx, etc. (simple examples, it may have a little complex relation).
I have seen that D2 and DX have a close correlation, how can I use this to relate it with D1 OR what conclusion can I draw from it? How can I analyze these three variables together? Whether I can do something like D1 Vs (D2 and DX)?
The dataset looks like this: (but having large number of data)
D1 D2 DX
3.3 3.0 3
1.12 0.83 1
4.4 6.44 6
2.5 11.9 15
0.79 2.19 3
3.81 9.63 12
2.4 4.9 7
2.56 13.2 13
1.97 5.26 5
4.50 3.68 4
4.72 5.21 6
0.89 4.10 5
1.78 5.89 8
3.2 7.54 12
3.18 2.20 4


texts: Algorithms behind "smartlinks" for online traffic distribution

texts:
Several traffic networks offer "smart links", where you can send online traffic, and the traffic will be sent to the most profitable segment. These networks include Monetizer, Glize etc.

I.e. if you send IOS, mobile, Orange carrier in Spain, the traffic will primarily go to offer Y. If you send Android, Tablet in France it will go to offer X etc. I assume they optimize for earning per view/click. A fraction will be for testing, but the majority of the traffic will go to the optimal offer.

I wonder what the general workings of such algorithms are, and if there's a place (research papers, or books), I can learn more about this type of problem? I have a hard time finding anything directly related online, as it's not a very common problem in research papers, and a lot of the algorithms are being disclosed -- all though some mention using Bayesian statistics to optimize.

I started looking at multi-armed bandit problems, which I feel are very close to this type of problem. If you imagine the different offers are different "bandits", which bandit should you choose to maximize return. From this they can calculate a percentage weight per offer. Maybe initially they will all start at an even weight, and once there's enough data in the timespan, the weights will be shuffled (at a timed basis, i.e. every 30 min).

I think this is a very interesting problem, as I work in online marketing and also study computer science.

Thanks in advance.


texts: What is the upscaling factor in super resolution with deep learning?

texts:
I have been reading papers on single image super resolution (SISR) and I frequently encounter X3 upscaling factor, X4 upscaling factors.
Example: SRGAN mentioning x4 upscaling factor
It would be wonderful if anyone could explain it in simpler words. Thanks


texts: Why is this series of house prices being thrown into one bin with plt.hist?

texts:
I have a series of houseprices that looks like this:

0    410000
1    610000
2    579950
3    559950
4    404950
Name: price, dtype: int64

And using plt.hist(df.price) I get the below histogram, which is not useful. What is causing this? Am I missing something here? I want a range of prices to be binned together and their frequencies to be plotted.


texts: TensorFlow Time Series Tutorial Enhancement Gone Wrong

texts:
I’ve been following this time series tutorial for Tensorflow…

https://www.tensorflow.org/tutorials/structured_data/time_series

And it was going good, and seemed to work ok.  I substituted with my own dataset (about 1.1m records with 4 features) and it also seemed to work well, but memory was getting REALLY tight, so I thought I would try and implement the improvement mentioned at the bottom that said…

  In addition, you may also write a generator to yield data (instead of
  the uni/multivariate_data function), which would be more memory
  efficient. You may also check out this time series windowing guide and
  use it in this tutorial.

This kind of made sense, as it seems to produce a sliding time window with single steps so the data becomes HUGE.  After reading through this…

https://www.tensorflow.org/guide/data#time_series_windowing

It looked like I could use tensor flow to do the windowing, which I assumed would get executed dynamically therefore not much memory would be used.  After reading, I came up with the following code to try and replace the multivariate_data function (functions are borrowed from the tutorial, though I changed the dense_1_step to get a single set of label features back instead of a window…

def make_window_dataset(ds, window_size=5, shift=1, stride=1):
  windows = ds.window(window_size, shift=shift, stride=stride)

  def sub_to_batch(sub):
    return sub.batch(window_size, drop_remainder=True)

  windows = windows.flat_map(sub_to_batch)
  return windows

def dense_1_step(batch):
  # Shift features and labels one step relative to each other.
  return batch[:-1], batch[-1:]

# get training samples (features and labels)
train_ds = make_window_dataset(tf.data.Dataset.from_tensor_slices(dataset[:TRAIN_SPLIT]), window_size=past_history+1, shift = 1, stride=1)
dense_labels_train_ds = train_ds.map(dense_1_step)

# get validation samples (features and labels)
val_ds = make_window_dataset(tf.data.Dataset.from_tensor_slices(dataset[TRAIN_SPLIT:]), window_size=past_history+1, shift = 1, stride=1)
dense_labels_val_ds = val_ds.map(dense_1_step)

# batch and shuffle training data
train_data_single = dense_labels_train_ds.cache().shuffle(BUFFER_SIZE).batch(BATCH_SIZE).repeat()
val_data_single = dense_labels_val_ds.batch(BATCH_SIZE).repeat()

But for some reason it's all gone pear shaped for the following reasons...

The training loss and validation loss are now NOTHING alike and in fact I don’t think the net is even learning properly anymore
The memory usage seems much better UNTIL you execute the fit() and then it just blows up about the same anyway.
I get the warning ‘Sets are not currently considered sequences, but this may change in the future, so consider avoiding using them.’

I’m sure I screwed something up here, but to me it looks like it's doing what it should be doing.  I'm still on a steep learning curve with this, but if anyone with more experience could point me in the right direction I would be very grateful.

Thanks

Ray


texts: Why would 2 sets of similar training samples take significantly longer to train?

texts:
I've built a fully-connected feed-forward neural network to recognize handwritten digits. I used MNIST and another very similar dataset (containing Arabic digits - same training set and test set count as those of MNIST).

For a network with exactly the same architecture (1 hidden layer, same learning rate, same weight sampling), I tested on a sub-sample that used 100 images for training. The test accuracy was very low, but what I noticed was that for the Arabic digits, the network took significantly longer to train (double the amount of time than for English digits - 33ms and 15ms, respectively).

What could be the reason for this?


texts: Data brokers for market-related data, how to choose, where to look for?

texts:
Situation

I am doing a data science project for my client. Due to NDA I can't disclose anything to the nature of the project, but let's imagine that they manufacture and sell slow cookers.

I faced the need to get the dataset which will cover all of the 'slow cooker' products in the selected market. The information we want is the list of features of each product. For example a record should have product name and brand + capacity, material, color, cord length etc., everything you could see on a product listing page in something like Amazon. The more detailed it is the better.

Beyond that we want either a comprehensive set of product review texts from leading ecommerce platforms. If not available - at least average (also min and max) customer sentiment of such reviews.

Issue

I already did a prototype research by just doing a web-scraping of one of eCommerce web-sites. The issue is that:

It took quite a while
It is not really comprehensive - I want to cover all leading eCommerce platforms.
Scraping more platforms will take a lot of time, and normalizing and merging scraping results will be a big pain.

On the other hand, client is OK to just pay some money to buy clean dataset from company which already did that - data brokers.

But the issue I faced is that even though I spent a lot of time googling data brokers I wasn't able to come up with a good list and clearly understand if these companies actually sell the kind of data I want. And honestly if you google 'data broker' absolute majority of results will give you some angry articles from newspapers about 'companies who know more about you than google', not the links to actual companies and the description of what kinds of data they provide.

Question

Bottom line - is there any advice on how to approach the issue of looking for a good data broker and choosing it? Is there any good resource which covers market leaders in this area with a comprehensive information on which kinds of data they sell?

Extra question: I remember some time ago Microsoft launched so-called Azure Data Mart which supposed to be a one-stop shop for datasets sold by various companies, but seems like the idea dies without further development. I wonder if anyone knows of such successful data marts which can be used?

Why I didn't post it on opendata

Answer is simple - I am not looking for open (free) data. Our clients are OK to pay for it. At the same time, OpenData community requires questions to cover only OpenData.


texts: Understanding outputs from ANN and how to improve validation loss

texts:
I apologise if this is a bit long winded, but it was suggested by another user that I post.
I will start by saying that I am very new to the world of machine learning and deep learning. As such, the most important thing I am after is the understanding of what I am doing.
I am trying to build an ANN for binary classification.
I have a binary feature matrix in the form of N x D, where N is number of samples and D is number of features. In my dataset N has a max of ~ 2 million - but I run my testing on ~ 500k due to the time it takes to run (even when utilising my GPU). If I get a promising validation curve after testing, I will run on the full dataset to verify. D in my dataset is 5. Thus, I have a feature matrix of the form 500000 x 5. A snippet below:
[[0 1 0 1 1]
 [0 1 1 0 1]
 [0 0 1 0 1]
 [1 1 0 1 1]
 [1 1 0 0 0]
 [0 1 1 0 1]
 [0 1 0 0 0]
 [1 1 0 1 1]
 [0 0 0 1 1]
 [1 0 0 0 1]]

I have a target matrix in binary form, snippet below:
[1 0 1 1 0 0 1 0 1 1]

Based on my understanding, for binary classification, the Input layer should be the same as D, and your output layer should have 1 node and it should have a sigmoid activation function.
Thus, I have taken this approach. Now, I also understand that machine/ deep learning is a lot of experimentation and so I have gone through many different iterations of changing the number of hidden layers, as well as changing the number of nodes per hidden layer - all to no seemingly obvious benefit. I have also played around with the following: learning rate (for Adam optimiser), train to test ratio (currently at 0.33), random state variable (when splitting the dataset - currently at 42), batch size (currently at 128), epochs (currently at 50). All of this experimentation with these variables still lead to a high validation loss, as I will show further below.
Now, for my code. Below is my code to split the data in to train and test.
train_to_test_ratio = 0.33
random_state_ = 42

# split the data into train and test sets
# this lets us simulate how our model will perform in the future
x_train, x_test, y_train, y_test = train_test_split(X, Y, test_size=train_to_test_ratio, random_state=random_state_)
N, D = x_train.shape

Below is my code to to build the model. You can see I currently have 3 hidden layers. I have played around this a lot - from 1 hidden layer to 10 with varying nodes per layer. The best I have found is the below 3 hidden layers. The learning rate is currently 0.001 which creates the best loss curve - anything bigger is too high.
learning_rate = 0.001

# build the model
i = Input(shape=(look_back_period,))
x = Dense(8, activation="relu")(i)
x = Dense(16, activation="relu")(x)
x = Dense(32, activation="relu")(x)
#x = Dense(32, activation="relu")(x)
#x = Dense(64, activation="relu")(x)
#x = Dense(128, activation="relu")(x)
#x = Dense(64, activation="relu")(x)
#x = Dense(32, activation="relu")(x)
#x = Dense(16, activation="relu")(x)
#x = Dense(4, activation="relu")(x)
x = Dense(1, activation="sigmoid")(x)
model = Model(i, x)
model.compile(optimizer=Adam(lr=learning_rate), loss="binary_crossentropy", metrics=["accuracy"])

I then train the model:
# train the model
# trains on first half of dataset, and tests on second half
b_size = 128
iterations = 50
r = model.fit(x_train, y_train, validation_data=(x_test, y_test), epochs=iterations, batch_size=b_size)

Here is the validation curves:

Here is the accuracy curves:

As you can see the validation loss is very high, and the accuracy is not too high, which to me the model is failing.
What do these outputs mean? In the sense of with all the experimentation I still cannot get the validation loss low with high accuracy. This to me is that the model is incorrect. But what would be the correct model? Is there any advice that can be given to better understand how to move forward and build a better model?


texts: Is Keras a fancy tool or is it actually helpful in research or both?

texts:
I’m in second year of my PhD in a environmental-science laboratory and there aren’t many experts on machine learning here.

In one part of my research I should use a multilayer perceptron to build a model from a series of inputs that I determined before. So for this part I suggested to use a deep learning framework, specifically Keras with Tensorflow or CNTK, but in the lab my supervisors told me that for a real application they don’t want to depend of a framework from Google or Microsoft and they prefer to develop they own tools that won’t change in the time if there’s for example an upgrade or something like that. One researcher created a neutral net and he obtained good results, but when we are interested in make some fine tuning, his code is more complex to update or to test new things like an dropout for example.

I started to work with those frameworks having in mind the reproducibility because I only have to use the functions from Keras and define the parameters. I think other researchers who want to test/use my model should use the configuration I suggest and don’t try to understand the code I provide. So I obtained some good results and am now in the phase of fine tuning, but the work of the other researcher has a slightly better results but after a regularisation step that I didn’t try yet.

So this researcher told me that I should abandon my work with Keras and start using his code. He also told me that Keras is just some fancy tool and in research is not required to have a very precise model, just good results and use the code that works.

So my question is this: Is it right what this researcher told me? Is Keras actually used in research? If there are some frameworks that are developed, why not use them instead of rediscovering the wheel, am I right?


texts: How do I draw the following Keras model?

texts:
I am new to deep learning models, and I am trying to figure out the best way to draw a visual representation of the following model. I found a few tools that allow you to draw a model, but I am confused about what the 'best' way to draw this particular model would be.

Any suggestions or recommendations would be great.
Thanks!


texts: How to do embedding for nested dictionary with varying size?

texts:
I'm working on an RL task in which the agent needs have some observation. Instead using images, I want to use available information of the environment as the observation. The information regarding the environment is about the info of the objects that exist in the environment and is stored as a list of nested dictionary:
[
{'name': 'TennisRacket_de7db8d2',
  'position': {'x': -1.667, 'y': 0.0, 'z': 0.818},
  'rotation': {'x': 90.0, 'y': 29.999939, 'z': 0.0},
  'cameraHorizon': 0.0,
  'visible': False,
  'receptacle': False,
  'toggleable': False,
  'isToggled': False,
...},
{'name': 'Laptop_f5306a34',
  'position': {'x': -1.39652, 'y': 0.8289251, 'z': -1.89688826},
  'rotation': {'x': 0.0, 'y': 19.242403, 'z': 0.0},
  'cameraHorizon': 0.0,
  'visible': False,
...}
...]

So the task now is to do an embedding on this data. I have thought about putting encoding(eg. utf-8) for non-numerical characters into an vector, using averaged word2vec and doc2vec but they all seem not suitable for the following reasons:

for directly using encoding, the dictionary might contain different numbers of observed objects so the length of the dictionary will vary, but the observation requires a vector of the constant size and shape.

for averaged word2vec and doc2vec, it is highly likely the case that some of the objects has much more importance, so I think it is better to be able to differentiate them individually and I thing both averaged word2vec and doc2vec will aggregate everything together and hence lose the individual object's info.

Can someone suggest a embedding method that both preserves of the shape/size of the embedded vector regardless of the size of the input while keeping the information of individual objects? Thanks for replying.


texts: Is Stratification applicable to both Classification and Regression?

texts:
It is usually better if you have a not so large but balanced dataset and you are performing classification to apply stratification in order to split it in a training and testing datasets which are both balanced as well.

So there is a notion of having the training and testing datasets represent adequately the overall dataset.

Could you expand this notion to regression?

So the idea is that instead of just shuffling and splitting the dataset, you could group the targets in bins for example [0, 10] [10, 20] etc.
And then have the training and testing dataset be also an adequate representation of the whole dataset by having targets with all kinds of values. (Otherwise, you could end up leaving out some part of the range)

Makes sense? :)


texts:
I'm trying to develop a multi regression model to predict energy consumption during one day period.
X-set dimension is (10178, 52) and consist of 52-feature and Y-set dimension is (10178, 48) as output. I have used the following code:
xtrain, xtest, ytrain, ytest=train_test_split(X, Y, test_size=0.1)

in_dim = X.shape[1]
out_dim = Y.shape[1]
model = Sequential()

model.add(Dense(48*4, input_dim=in_dim, activation="relu"))

model.add(Dense(86, activation="relu"))
model.add(Dense(out_dim))
model.summary()
model.compile(loss="mean_absolute_error", optimizer="adam")
model.fit(xtrain,ytrain, epochs=100, batch_size=12,)

after compiling my model although my model's loss is very low but when I visualize my output the result is unsatisfying as follow:

any idea what I'm doing wrong?!? my initial guess is that since output dimension is high(48-dimension) compared to input dimension I need a lot more Data. or maybe I'm using wrong loss function or the model is too shallow. also it is noticeable that model's output at spark point is very poor.


texts:
I'm requesting data from a government body, and they asked me what format I want to receive the data in. This will be a table of about 400,000 rows and about 10 columns. My options are:

"comma or tab delimited ASCII, Microsoft Access database, Microsoft Excel file etc"

They also want to know the media by which I want to receive data. My options are:

"encrypted CD, DLT Tape, etc."

I'm used to working with JSON, and doing data analysis via Python and R. These data will have to remain confidential.

Any pointers?


texts: How does scaling affect Logistic Regression?

texts: I have searched a lot on the web for this question, but I never seem to find a consistent yet straight forward answer. Simply put, the question is: How exactly does scaling affect logistic regression? What should you expect from a logistic regression classifier when the data is scaled and when it is not? Is there any difference between min-max scaling and standard scaling in terms of logistic regression?


texts: What is the difference between Slow Feature Analysis (SFA) and a Moving Average?

texts:
I have started to read more about Slow Feature Analysis and I was wondering how SFA differed from simply taking a moving average?

The linked article suggests, "SFA is an unsupervised algorithm that learns (nonlinear) functions that extract slowly-varying signals from their input data.".

I understand that it is certainly more complicated than just taking a moving average on a signal, but I don't really understand what the benefit or purpose is of the learned non-linear functions.

What would be some typical applications of SFA?


texts: When to model a problem by using the Bayes' theorem?

texts:
I have a labeled training dataset where each observation has a sentence either in English or in French as its predictors and its label (target value) is whether this sentence is English or French. The test set includes again some sentences either in English or in French but without labels.

A friend of mine suggested that we should model this problem by using the Bayes' theorem since we have have some prior values (labeled observations in training set). I agree that this can work too but I cannot really understand his argument "we should model this problem by using the Bayes' theorem since we have have some prior values".

This is because in my mind every labeled observation can be considered as a prior value and every prior value can be considered as a labeled observation so you can also apply any machine learning classification algorithm e.g. decision trees) in these cases.

Is this right in general or at least for this specific problem?

Why Bayes' theorem modeling comes up as the best solution for the problem which I described above?


texts: Appropriate objective function and evaluation metric when I DO care about outliers?

texts:
I am reading these two pages:
xgboost documentation
Post on evaluation metrics
I have a dataset where I am trying to predict future spend at the user level. A lot of our spend comes from large spenders, outliers. So, we care about them. I am using XGBoost.
I have tried xgboost with objective reg:squarederror. This tended to underpredict a little. I then tried with reg:squaredlogerror and this resulted in predictions that under predict by much more than just using squarederror.
I have tried tuning with several differing hyper parameter combinations but none made as big a difference as changing the objective. So, I'm dwelling on the objective function and trying to understand if there's another one out there that would be worth a shot?
On the xgboost docs above, some of the out of the other regression objective options are reg:pseudohubererror as well as count:poisson.
There is no option, that I can see, for just MAE. If using an objective function less susceptible to outliers with rmsle took me further away from accuracy whereas rmse took me closer, would using MAE potentially be worth a shot? In this dataset, outliers are more important, but so are regular users.
What would be a good objective and evaluation metric?
Is MAE worth trying? If so, how? Looking at the docs above, I cannot see MAE as an option under regression parameters.


texts: Issues related to the code for ROI pooling from the feature map

texts:
I am trying to do ROI pooling on the feature map obtained from the VGG layers but I don't know how to code this layers. Can anybody help me out?

Here is my VGG layers:

model=Sequential()
    model.add(ZeroPadding2D((1,1),input_shape=(3,112,112)))
    model.add(Convolution2D(64, 3, 3, activation='relu'))
    model.add(BatchNormalization(momentum=0.99))
    model.add(ZeroPadding2D((1,1)))
    model.add(Convolution2D(64, 3, 3, activation='relu'))
    model.add(BatchNormalization(momentum=0.99))
    model.add(MaxPooling2D((2,2), strides=(2,2), dim_ordering="th"))

model.add(ZeroPadding2D((1,1)))
model.add(Convolution2D(128, 3, 3, activation='relu'))
model.add(BatchNormalization(momentum=0.99))
model.add(ZeroPadding2D((1,1)))
model.add(Convolution2D(128, 3, 3, activation='relu'))
model.add(BatchNormalization(momentum=0.99))
model.add(MaxPooling2D((2,2), strides=(2,2), dim_ordering="th"))

model.add(ZeroPadding2D((1,1)))
model.add(Convolution2D(256, 3, 3, activation='relu'))

`
The corresponding model is described in the picture below:


texts:
I'm using Tensorflow for an auto-tagging task on audio clips. The problem is actually a multilabel classification problem meaning that each clip can have multiple tags at the same time.

Regarding the model evaluation, I would like to be able to compute the AUC per clip and per tag. My problem is that AUC is basically a metric for binary classification and I don't know how it can be extended to a multilabel problem.

I have found many references of AUC computation on multilabel classification tasks, but none of these explain how to do it.

(An example)


texts: Why Keras Dense layer is expanding number of tensors in each layer

texts:
I have following model:

and I wonder why is the number of parameters different for e.g. dense_2477 and dense_2482? Both layers have the same amount of neurons so why do they provide different parameter numbers?


texts: How is the 'feature_importance_' value calculated in sklearn random forest regressor?

texts:
I have 9000 sample, with five features, and one output variable (all are numerical, continuous values). I used random forest regression method using scikit modules. I got a graph of the feature importance (using the function feature_importances_) values for each of the five features, and their sum is equal to one. I want to understand what these are, and how they are calculated mathematically.
Can someone please explain?


texts:
I just learned that regularizing the mean encoding reduce the leakage hence generalize better than mean encoding without it but I made 2 submissions with XGB in predict future sales competition on Kaggle  with the naive mean encoding method and got RMSE = 1.152 and with 5 folds validation and got RMSE = 1.154 which was a surprise for my. Can any one explain why this may happen ? also after making the kfolds regularization every item_id has multiple mean encoding one for each fold so should I leave the train set in this format or take their mean ? also when applying this to the test should I take the mean and apply it or what ?

I'm encoding the item_id with the target which is the number of sold items :

Naive method :

all_data['item_target_enc'] = all_data.groupby('item_id') ['item_cnt_month'].transform('mean')

Regularization :

kf = KFold(n_splits = 5 , shuffle = False)

for train_ind, val_ind in kf.split(all_data) :

train_data , val_data = all_data.iloc[train_ind], all_data.iloc[val_ind]

all_data.loc[all_data.index[val_ind] , 'item_target_enc'] = val_data['item_id'].map(train_data.groupby('item_id')['item_cnt_month'].mean())

all_data['item_target_enc'].fillna(all_data.item_cnt_month.mean(), inplace=True) # fill na that happened from values in validation not in the train with the global mean

Here is the encoding result with regularization for each item_id i got an array which was confusing for my to take the mean or not :


texts: DecisionTreeClassifier Integer Conditions, Integer Outcome Variable

texts:
Vague condition: "NumGoals >= 1.23"

Preferred condition: "NumGoals > 1".

Switched normalization off.

Code:

from sklearn.datasets import load_iris
from sklearn.tree import DecisionTreeClassifier, plot_tree
import matplotlib.pyplot as plt
import numpy as np

iris = load_iris()

clf = DecisionTreeClassifier()
clf = clf.fit(np.array(iris.data, dtype=int), iris.target)
plot_tree(clf)
plt.show()

Possible to treat integers as integers?


texts: Training models from sklearn using tf.distribute.MirroredStrategy

texts:
I want to distribute the training of a simple model, such as a support vector classifier like sklearn.svm.SVC() across some or all CPUs and GPUs on a single device. I have never utilized a GPU before and I'm confused as to how this works or if using tensorflow is even the right choice for this simple task. What i think I need to do is something like this:

import tensorflow as tf
from sklearn.svm import SVC
from sklearn import datasets
from sklearn.model_selection import train_test_split

strategy = tf.distribute.MirroredStrategy()

with strategy.scope():

    iris = datasets.load_iris()
    X = iris.data
    y = iris.target
    class_names = iris.target_names

    X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=0)

    classifier = svm.SVC(kernel='linear', C=0.01).fit(X_train, y_train)

The actual dataset I'm using is $\approx 3 \times 10^8$ training examples with 11 features. Does this code do what I think it does? If not, what would be the best way to go about this task? If so, is there anything that can be improved?

EDIT:

After doing some more googling I discovered that sklearn does not support GPU utilization. See this post:
https://stackoverflow.com/questions/41567895/will-scikit-learn-utilize-gpu

I'm still not sure how I can go about utilizing a GPU for simple ML models.


texts:
Lets say that you have panel data, of daily consumption of product p1 of 10000 individuals. The panel data is on a daily basis for only one month, this means that $t=1,...,30$.
The question is how do it ? What kind of model I to use and why.

My answer iss to have a two-level model:

$y_{i,t}=\beta_{i}x_{i,t}+\epsilon_{i,t}$

$\beta_{i}=\gamma_{i}z_{i} + \eta_{i}$

where $x_{i,t}$ would contain my time variant variables and $z_{i}$ will contain all my time invariant variables. My argument was, that by modeling it like this, I believe that I take control of the time dependency between the $y_{i,t}$ and $y_{i,t-1}$ by letting the $\epsilon_{i,t}$ also depend on time and also I capture the heterogeneity of the consumers. Where $\epsilon_{it}\sim N(0,\Sigma_{\epsilon})$ and $\eta_{i}\sim N(0,\sigma_{ \eta _{i}}^2)$

What do you think ?


texts: Could someone give an example of the application of Tf-idf with sparse data (lots of zeros) in sklearn? I am not quite sure where to insert the weight of Tf-idf and how to rightly obtain the weight. Without knowing this fully, I would not be able to well use the tool for prediction. Thank you.


texts: Reporting descriptive statistics in case of missing data

texts:
I have a dataset of survey responses with 10 variables. 4 of them were optional questions during the survey. Now for ~900 participants, I have missing values for the 4 optional questions, but these have informative data for the other questions. What is the best way to represent them in my descriptive analysis report?
Any leads are appreciated.
Thanks.


texts: What model is suitable for classification of a small data set?

texts:
I have a dataset that consists of 365 records, and I want to apply a classification model on it (binary classification).

As an output, in addition to the classification labels, I want to retrieve the classification confidence for each instance.

I don't know how to deal with such a case. Can I use, for example, linear classifiers (SVM, logistic regression) with this small dataset? Because, I want to retrieve the classification confidence as well.

I read that decision trees can be a good classifier for small datasets, but how can I retrieve the classification confidence with it?

The dataset consists of tweets, each classified as positive or negative (from a sentiment perspective), and my feature vector consists of 2400 features (combination between word2vec embeddings and other features).

Also, do you recommend me to use word2vec embeddings with such a small dataset? I think the classifier can't learn something from them using small dataset.


texts:
I have a json array f below format

[{
    'Address': 'xxx',
    'Latitude': 28. xxx,
    'Longitude': 77. xxx,
    'reached': False
}, {
    'Address': 'yyy',
    'Latitude': 18. yyy,
    'Longitude': 73. yyy,
    'reached': False
}]

i want to convert into dataframe. if the column name is same it should have (Address_0, Address_1 etc) and should be side by side, not below. How can i do this?


texts: Insignificant input variable but high R2

texts:
Can a linear regression model with a single input variable x with x not significant but the model has a high R2?

Can this exist? If yes, what are the reasons?


texts: How to save Tensorflow predictions to data frame?

texts:
I am new to Tensorflow. I have trained Tensorflow model, but I need to take model predictions and add them to my original test set as a column. How can I do that?

def model(self, layers_dims, X_train, Y_train, X_test, Y_test, learning_rate=0.00001,
          num_epochs=1000, print_cost=True):
    """
    Implements a three-layer tensorflow neural network: LINEAR->RELU->LINEAR->RELU->LINEAR->SOFTMAX.

    Arguments:
    X_train -- training set, of shape (input size = 12288, number of training examples = 1080)
    Y_train -- test set, of shape (output size = 6, number of training examples = 1080)
    X_test -- training set, of shape (input size = 12288, number of training examples = 120)
    Y_test -- test set, of shape (output size = 6, number of test examples = 120)
    learning_rate -- learning rate of the optimization
    num_epochs -- number of epochs of the optimization loop
    minibatch_size -- size of a minibatch
    print_cost -- True to print the cost every 100 epochs

    Returns:
    parameters -- parameters learnt by the model. They can then be used to predict.
    """

    ops.reset_default_graph()  # to be able to rerun the model without overwriting tf variables
    tf.set_random_seed(1)  # to keep consistent results
    seed = 3  # to keep consistent results
    (n_x, m) = X_train.shape  # (n_x: input size, m : number of examples in the train set)

    n_y = Y_train.shape[0]  # n_y : output size
    #print('Ytrain shape', Y_train.shape)
    costs = []  # To keep track of the cost

    # Create Placeholders of shape (n_x, n_y)
    X, Y = self.create_placeholders(n_x, n_y)

    # Initialize parameters
    parameters = NN_predict_trading_decisions.initialize_parameters(layers_dims)

    # Forward propagation: Build the forward propagation in the tensorflow graph
    ZL = NN_predict_trading_decisions.forward_propagation(X, parameters)

    # Cost function: Add cost function to tensorflow graph
    cost = NN_predict_trading_decisions.compute_cost(ZL, Y)

    # Backpropagation: Define the tensorflow optimizer. Use an AdamOptimizer.
    #optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cost)
    optimizer = tf.train.GradientDescentOptimizer(learning_rate).minimize(cost)

    # Initialize all the variables
    init = tf.global_variables_initializer()

    # Start the session to compute the tensorflow graph
    with tf.Session() as sess:

        # Run the initialization
        sess.run(init)

        # Do the training loop
        for epoch in range(num_epochs):

            epoch_cost = 0.  # Defines a cost related to an epoch
            current_cost = sess.run([optimizer, cost], feed_dict= {X: X_train, Y: Y_train})
            epoch_cost +=  current_cost[1]

            # Print the cost every epoch
            if print_cost == True and epoch % 100 == 0:
                print("Cost after epoch %i: %f" % (epoch, epoch_cost))
            if print_cost == True and epoch % 5 == 0:
                costs.append(epoch_cost)

        # plot the cost
        plt.plot(np.squeeze(costs))
        plt.ylabel('cost')
        plt.xlabel('iterations (per tens)')
        plt.title("Learning rate =" + str(learning_rate))
        plt.show()

        # lets save the parameters in a variable
        parameters = sess.run(parameters)
        print("Parameters have been trained!")

        # Calculate the correct predictions
        correct_prediction = tf.equal(tf.argmax(ZL), tf.argmax(Y))

        # Calculate accuracy on the test set
        accuracy = tf.reduce_mean(tf.cast(correct_prediction, "float"))

        print("Train Accuracy:", accuracy.eval({X: X_train, Y: Y_train}))
        print("Test Accuracy:", accuracy.eval({X: X_test, Y: Y_test}))

        return parameters


texts: Problem with weights having the same shape[0] during forward propagation and preventing the dot product from working

texts:
I'm trying to implement forward propagation in my neural network but it doesn't seem to work and I suspect it is because the number of neurones from one layer to the next are not the same.
shape of X :  (134, 3072)
shape of W1 :  (3072, 101)
shape of b1 :  (1, 101)
shape of Z1 :  (134, 101)
shape of A1 :  (134, 101)
shape of W2 :  (3072, 102)
shape of b2 :  (1, 102)

The first step is to get Z1 = np.dot(X, W1) + b1, then A1 = activation_function(Z1).
The second step is where everything falls apart :
I want to get Z2 = np.dot(A1, W2) + b2, however the dot product is invalid because we have shape (134, 101) with (3072, 102).
Should I truncate W2 to be of the shape (101, 102)? Otherwise, what should I do?


texts: Train Test Split for overlapping samples

texts:
I'm doing a classification of time sequenced sensor data (in Python), where I'm segmenting the sensordata into samples, with a certain window-length (e.g. 3 seconds). However, the samples are also overlapping each other. For example, the first sample is 0s $\rightarrow$ 3s, second is 2.7s $\rightarrow$ 5.7s...

I'm wondering now, how I can do a proper train test split for this samples. Right now, I see two ways to do this:

Split the samples without shuffling them first and drop the sample "at the border", to avoid overlap between train and test data. However, this is not optimal because I would like to have samples from all over the dataset for both the training and the test set. Otherwise I would just test the classifier on a sequence of the data, that might be very different from the sequence used for training.
Shuffle the samples first and then split them into training in test. This would result in an overlap between training and test data and would therefore produce overly optimistic results.

Does anyone have another idea how this could be done?


texts: I used Stargazer in Python. How to convert output to LaTex?

texts:
Hello I typically use R but is there a way how to use stargazer in python

For example when doing an arima model

Model = SARIMAX(data, order = (1, 0, 0), trend = “c”
model_results = Model.fit()
Print(model_results.summary())

I get output table - is there a way how to convert it to latex


texts:
Say, you have data on customers' purchases. Data looks like

pID, date, town, price, place, item
100, 0203, True, 1.23, New York, Coca-Cola
100, 0203, True, 0.99, New York, bread
101, 0501,True, 1.43, Idaho, pen
102, 0502, False, 1.59, Nevada, eggs
102, 0502, False, 0.99, Nevada, Coca-Cola

And you have b illions of
So, we have a set of purchase receipts. We need to figure out that $0.01 discount on soft drinks increases the sales volume by a factor of ten. That people purchase soft drinks with chips. That it is important to have a (lighter OR matches) if you sell cigarettes.
What would be a good tool to classify such receipts?


texts: How to verify that a trained neural network is trustworthy?

texts:
Currently I'm looking into possibilities to prove that a trained neural network (based on a regression task) performs well on the whole sample space based on statements made from test set metrics. In other words, I want to make sure that there is no sample or region of samples out there, where the network performs badly on, assuming that the sample space is finite.
Are there any well established approaches for this?


texts: Scikitlearn - TfidfVectorizer - how to use a custom analyzer AND still use token_pattern

texts:
The docs state that token_pattern is only used if analyzer == 'word':

token_pattern : string

    Regular expression denoting what constitutes a “token”, only used if
    analyzer == 'word'. The default regexp selects tokens of 2 or more
    alphanumeric characters (punctuation is completely ignored and always
    treated as a token separator).

Below is my desired pipeline:

analyzer = TfidfVectorizer().build_analyzer()
stemmer = SnowballStemmer('english')

def processed_words(doc):
    return (stemmer.stem(w) for w in analyzer(doc))

vec = TfidfVectorizer(analyzer=processed_words, strip_accents='unicode',
            stop_words='english', token_pattern=r'\b[^_\d\W]+\b')

If I include analyzer=processed_words, then I lose the ability to cut out the thousands of features that are numbers, underscores, and whatever other invalid character sequences specified in the regex.

Is there no way to achieve both stemming and token_pattern matching at the same time? Must I loop through all my documents ahead of time and rejoin the split documents after filtering using regex?

How about if I want to apply stemming AND lemmatization using two analyzers (include also WordNetLemmatizer()) -- how does one handle that?


texts: Different number of features in train vs test

texts:
I'm doing the titanic exercise on kaggle and there is a categorical Cabin attribute that has a lot of different strings: C41, C11, B20 etc. (about 100).

To be able to train my model I'm converting it to numerical attributes (using pandas get_dummies()). So in the end I get 100+ attributes.

On the test dataset however, there are less cabins, so I'll end up with fewer attributes.

I did something like this to make them equal (create columns that are in the training set and delete those that aren't):

for column in X.columns:
    if column not in X_test.columns:
        X_test[column] = 0

for column in X_test.columns:
    if column not in X.columns:
        X_test.drop([column], axis=1, inplace=True)

but I know it is not a good thing. So how else should I approach it?

I tried removing the cabin column altogether but my model performs better on test data with that column.


texts:
I have a dataset that contains information of pets (Breed,color,age,some text descriptions) and images of that pet. One pet can have more images than others.

I want to combine these images somehow to generate a single 1x255 feature so that I can merge it with the non-image features and use for my model.

One way I can think of is to put each image of a pet through a pretrained model to get a 1x255 output, then averaging out all of the outputs to get a single feature of that pet. But I havenät seen any paper/people doing the same to backup my intuition...

I'm new to data science, so if you have any thoughts just let me know. Thanks in advance.


texts: References for longitudinal data analysis

texts:
So my goal is to study longitudinal data (data in time series) by applying some data mining techniques. Ultimately I want to be able to "predict" outcomes. For example, a study of patients along the years/months (repeated measures) can be used to predict some diseases using things like logistic regression, neural networks, etc..

I'm using R. However I'm having a hard time finding good references on this subject. I have a mathematic background, even though I've applied some data mining techniques before, I've never applied those to time series.

If someone can recommend me some good theoretical references or hands-on examples in R, that would be greatly appreciated. Also if you know a relevant article on time series analysis, that would be great too. Thanks in advance.


texts: I want to use deep learning for regression. However, the number of training samples is not large. In image processing, some new samples are generated on the basis of initial data through tasks like rotation to increase the number of data sets. Can I know if there is any algorithm to generate more data based on current samples (data augmentation for regression)?


texts: How much imbalance in a training set is a problem?

texts:
In a simple binary classification problem, at what point does majority class to minority class become significant become significant? Intuitively, I would expect a 3:1 ratio to not be an issue, maybe not even a 10:1 ratio. But a 100:1 ratio certainly does require some action. What might be this cutoff?
As a followup, what might be come potential solutions beyond undersampling and oversampling?


texts: What is the difference between a (dynamic) Bayes network and a HMM?

texts:
I have read that HMMs, Particle Filters and Kalman filters are special cases of dynamic Bayes networks. However, I only know HMMs and I don't see the difference to dynamic Bayes networks.

Could somebody please explain?

It would be nice if your answer could be similar to the following, but for bayes Networks:

Hidden Markov Models

A Hidden Markov Model (HMM) is a 5-tuple $\lambda = (S, O, A, B, \Pi)$:

$S \neq \emptyset$: A set of states (e.g. "beginning of phoneme", "middle of phoneme", "end of phoneme")
$O \neq \emptyset$: A set of possible observations (audio signals)
$A \in \mathbb{R}^{|S| \times |S|}$: A stochastic matrix which gives probabilites $(a_{ij})$ to get from state $i$ to state $j$.
$B \in \mathbb{R}^{|S| \times |O|}$: A stochastic matrix which gives probabilites $(b_{kl})$ to get in state $k$ the observation $l$.
$\Pi \in \mathbb{R}^{|S|}$: Initial distribution to start in one of the states.

It is usually displayed as a directed graph, where each node corresponds to one state $s \in S$ and the transition probabilities are denoted on the edges.

Hidden Markov Models are called "hidden", because the current state is hidden. The algorithms have to guess it from the observations and the model itself. They are called "Markov", because for the next state only the current state matters.

For HMMs, you give a fixed topology (number of states, possible edges). Then there are 3 possible tasks

Evaluation: given a HMM $\lambda$, how likely is it to get observations $o_1, \dots, o_t$ (Forward algorithm)
Decoding: given a HMM $\lambda$ and a observations $o_1, \dots, o_t$, what is the most likely sequence of states $s_1, \dots, s_t$ (Viterbi algorithm)
Learning: learn $A, B, \Pi$: Baum-Welch algorithm, which is a special case of Expectation maximization.

Bayes networks

Bayes networks are directed acyclical graphs (DAGs) $G = (\mathcal{X}, \mathcal{E})$. The nodes represent random variables $X \in \mathcal{X}$. For every $X$, there is a probability distribution which is conditioned on the parents of $X$:

$$P(X|\text{parents}(X))$$

There seem to be (please clarify) two tasks:

Inference: Given some variables, get the most likely values of the others variables. Exact inference is NP-hard. Approximately, you can use MCMC.
Learning: How you learn those distributions depends on the exact problem (source):

known structure, fully observable:  maximum likelihood estimation (MLE)
known structure, partially observable: Expectation Maximization (EM) or Markov Chain Monte Carlo (MCMC)
unknown structure, fully observable: search through model space
unknown structure, partially observable: EM + search through model space

Dynamic Bayes networks

I guess dynamic Bayes networks (DBNs) are also directed probabilistic graphical models. The variability seems to come from the network changing over time. However, it seems to me that this is equivalent to only copying the same network and connecting every node at time $t$ with every the corresponding node at time $t+1$. Is that the case?


texts: features to help distinguish between document images

texts:
we are trying to build a model to classify different types of documents as the first step in our pipeline (final goal is to read all the text). Currently we use ImageNet to extract the features and then pass it through a custom model which includes an LSTM layer and a couple of FC layers. The model typically falters in the "same class" categories. For e.g. between one govt. issued card vs the other like shown below

as you can see both look different to humans but kinda have the same dense text in the center of the image. There's false positives across both these categories.

What additional features can we use along with just the ones extracted from a pre-trained model ? (we understand ImageNet wasn't trained on such images but the features it extracts, along with a simple LSTM+DENSE model gives us an F score of ~80%). We can try them all and use PCA to ascertain the best mix.

Here's what we tried (along with ImageNet features)

single dimension histogram (both color and grayscale images)
multi dimensional histogram (3 vectors that we stacked)
this just drives the overall accuracy lower.

Most documents we read online pertain to certain domains where domain specific features can be extracted. Not sure what we do here .. please guide


texts: Dividing a dataset to parallelize machine learning training on the cloud

texts: I'm very new to machine learning. I am doing a project for a subject called parallel and distributed computing, in which we have to speed up a heavy computation using parallelism or distributed computing. My idea was to have a dataset divided in equal parts, and for each subset to have a neural network to be trained on a separate machine in the cloud. Once the models are trained, they would be returned back to me and somehow combined into a single model. I am aware of federated learning but it doesn't quite fit my scenario of actually sending and dividing the dataset into the cloud. Does someone know any feasible approaches (maybe a variant of federated learning) of how one would do this?


texts: How to reapply rescaling when you want to predict your data


texts: Machine Learning models in production environment

texts:
Lets say a Model was trained on date $dt1$ using the available labeled data, split into training and test i.e $train_{dt1}$, $test_{dt1}$. This model is then deployed in production and makes predictions on new incoming data. Some $X$ days pass, and there is bunch of labelled  data that is collected in between $dt1$ and $dt1 + X$ days, lets call it $Data_x$.  In my current approach, I take  random samples out of  $DATA_x$ (take for e.g 80/20 split) ,

So,
$80\%$ of $DATA_x$ = $train_x$ (new data used to fine-tune the existing model trained on $dt1$)
$20\%$ of $DATA_x$ = $test_x$ (new data added to the $test_{dt1}$)

This process of fine-tuning repeated as time passes.

By doing this I get an ever expanding test set, as well as I prevent retraining the whole model (essentially I can throw away the old data as the model has learnt from it). The new model generated is just a fine-tuned version of the old one.

I have some questions, regarding this approach:

Are there any obvious drawbacks in doing this?
Would the model ever need to be completely retrained (forgetting everything that was learnt before, and training the model with new train/test splits ) after some time or can the approach I described above continue indefinitely ?
What should be the condition for swapping the existing deployed model with the newly fine-tuned model ?


texts: Why does an imbalanced data set badly effect distance measures like Mahalanobis?

texts:
I'm relatively new to data science and I am struggling to understand why the Mahalanobis distance (or any other distance measure) applied to an imbalanced data-set becomes inaccurate. I have a data set that consists of three classes A, B and C. There are 100 observations for class A, 60 for class B, and 20 for class C. When I calculate the Mahalanobis distance between each class, the results do not appear consistent with my PCA (principal component analysis) plot. In the PCA plot, class C is the most separate class; however, the Mahalanobis distance does not reflect this.
For balanced data sets, i.e., where classes A, B and C have the same number of observations, this has never been an issue. The Mahalanobis distance has always quite accurately reflected the results of PCA for balanced data.
I have read some similar questions and answers on here about why imbalanced data must be handled carefully for classification algorithms, but is this the same for distance measures? From what I can tell the Mahalanobis distance doesn't explicitly depend on sample size.
Therefore, I ask why does this measure lose reliability for imbalanced data?


texts: Orange Feature Construction widget: Inability to see formulas while file sharing

texts:
Our team is sharing Orange "work bench" files across team members. Unfortunately, we find the Feature Construction widget does not display code consistently (nor is there any "x" to indicate an error).
1. In 2 versions of 3.4.1, we could share the Feature Construction widget
2. However, Feature nodes created in version 3.4.1 could not be viewed in 3.13 or the standalone Orange.
An image of the original coder's screen - followed by the recipient's screen - is shown. How do we package the program appropriately to share across users who may have multiple versions of Orange?
Generally, do you recommend that data manipulation like feature engineering be done in Python instead of this widget?
Thanks, Layne


texts: BERT classifier with Ktrain API is unable to predict new data

texts:
I have trained a classifier for sentiment analysis using BERT architecture.
I am able to train the classifier and I am getting a validation accuracy of 87%. But whenever I feed in test data, or some simple sentence like "What an amazing movie", "I would love that book", etc, the model predicts the class for one and says list index out of range for other.
I tried to find if there is a bug in my code as well.

(x_train, y_train), (x_test, y_test), preproc = text.texts_from_csv(TWITTER,
                                                                       preprocess_mode='bert',
                                                                       text_column = 'text', label_columns = ['target'])

model = text.text_classifier('bert', (x_train, y_train), preproc=preproc)
learner = ktrain.get_learner(model,train_data=(x_train, y_train), val_data=(x_test, y_test), batch_size=6)

learner.fit_onecycle(2e-5, 1)

predictor = ktrain.get_predictor(learner.model,preproc)

predictor.predict(['I am very happy to meet you!'])

Error that I am getting

IndexError                                Traceback (most recent call last)
<ipython-input-28-82b8da552189> in <module>()
----> 1 predictor.predict(['I am very happy to meet you!'])

1 frames
/usr/local/lib/python3.6/dist-packages/ktrain/text/predictor.py in <listcomp>(.0)
     56                 preds = np.squeeze(preds)
     57                 if len(preds.shape) == 0: preds = np.expand_dims(preds, -1)
---> 58         result =  preds if return_proba or multilabel or not self.c else [self.c[np.argmax(pred)] for pred in preds]
     59         if multilabel and not return_proba:
     60             result =  [list(zip(self.c, r)) for r in result]

IndexError: list index out of range

My problem is that if I have any bugs in my code, then I should get this error every time I execute the predict method. Say I have 100 new test points, I am getting this error only for 30 to 40 test points and the remaining are classified properly. I tested this theory by feeding in tweets one tweet at a time. But I do not understand why is this happening.


texts:
In the paper "Discovery of frequent episodes in event sequences" by Mannila et al., a method for finding frequent episodes in an event sequence if a class of episodes and a sequence of events is given.

It is not clear to me how to find episodes in the first place. I would like to know if there are any tools, algorithms, or published work that can be used to this end, or which describes a method to find episodes in a given stream.


texts:
I'm a beginner in neural networks and currently I'm exploring the word2vec model. However I'm having a tough time to understand what the feature matrix exactly is.

I can understand that the first matrix is a one-hot encoding vector for a given word, but what does the second matrix signify? More specifically, what does each of those values (ie. 17, 24, 1 etc) mean?


texts: Siamese network do not compare pictures correctly

texts:
I have a siamese network trained for recognizing products.
Some of these products are recognized properly but some of theme not.

Images in each class are similar (For testing this architecture I had only one image per product so after argumentation there is 20 similar images per products) so it should not be a problem to identify the right one.

I am not sure if my code is working correctly.

sample = imread('C:/Datasets/Products/test_img.jpg')
sample = sample/255.
sample = np.expand_dims(sample, axis=0)

scores = list()
labels = list()

index = 0

for image in x_test:
    image = np.expand_dims(image, axis=0)
    scores.append(model.predict([image , sample])[0])
    labels.append(y_test[index][0])
    index +=1

scores = np.array(scores)
labels = np.array(labels)

index = np.argmax(scores)
label_ = labels[index]
print( 'IMAGE is {} with confidence of {}'.format(label_, scores[index][0]))

Images to x_test and y_test are loaded by below function

def loadimgs(path):
    X=[]
    y = []
    images=[]

    for product in os.listdir(path):

        prod_path = os.path.join(path, product)

        for img in os.listdir(prod_path):
            image_path = os.path.join(prod_path, img)
            image = imread(image_path)
            images.append(image)
            y.append(product)

    y = np.vstack(y)
    X = np.stack(images)

    return X,y


texts: Finding the best "depth" of ICD9 codes with pseudo-hierarchical clustering

texts:
Here is a common problem in health care modeling.  Did I just invent a new algorithm or has someone already thought of this?

The goal is to find the most homogeneous partition of patients by medical costs using ICD9 codes.  There are 13,000 individual codes in the data set, so using the full code results in many only having a few observations.

ICD9 codes are in a nested hierarchical structure.  For instance, all infectious diseases are 001-139, one particular disease is Cholera (ICD9 001), and this can have several other suffixes which further specify the illness.  This is a "drill down" so to speak.  001.0 and 001.1 are different types of the same disease.

The goal is to find the best level of detail for the data set.  For example, say that these codes are binary for simplicity, and are only 3 digits long.  Then the only possible codes are 000, 010, 011, 001, 111, 110, 101, 001.

One way of searching through all possible substrings of these would be to use a decision tree.  The x matrix would be all substring combinations, and the response would be the average costs for that particular code.

X would have a column for substring 0, 01, 010, 011, etc.  Each row of X would have 2^3*3 = 24 columns.  Each row would represent a single code.

The top 3 rows of X would look like this.  The response Y would be the average dollar amount of healthcare costs for that ICD code.

Once the data is in this format, a decision tree (or other model) could be used in order to determine which substring prefixes should stay in the model.  Because different ICD codes have different number of patients, this would be used as a weight vector.  The model could be tuned to allow deeper trees (more digits).

Is there a name for this already?
Thanks


texts:
I have a dashboard in tableau with a map, a bar chart (and a couple of other charts).
The bars on the bar chart represent a city.  I would love it if I could click on a bar and have the city on the map be selected.  (not filtered I want to be able to see the other cities on the map)  When the city is selected on the map a bunch of actions are triggered.
Is this possible?  I cannot for the life of me see how.


texts: Help with backpropagation equations for a simple neural network with Sigmoid activation

texts:
I'm trying to create a simple neural network with 2 input neurons, 1 hidden layer with 3 neurons and an output layer with single neuron and to train it for XOR problem.
 As per my understanding of neural nets, I'm planning to have in each neuron a weighted linear function of inputs and a Sigmoid activation function. So for example, the first input neuron with input x will compute c=ax+b, a being a parameter and b being its bias. Then it will feed
c to Sigmoid function to compute z=1/(1+exp(-c)) which will be its output. Please correct me if I'm wrong in any of this.

Now, I'm trying to follow this article to create back-propagation differential equations. It says that dc = z * (1-z) * dz where dz is the gradient of the green edge and then da = x * dc where da is the gradient computed in this iteration of backpropagation.

My question is that there are three zs (one corresponding to each outward edge) and their value(weight) will be identical (z=1/(1+exp(-c))). And I assume, then all values and gradients in the three neurons in hidden layer will also be identical. Then what's the point of having three neurons in hidden layer? What causes the randomness which can lead to three different gradients across each of the three green edges? And if they actually are different, how can I compute dc which is simply z * (1-z) * dz but there are three zs and three dzs?


texts: I was wondering since CNNs have dominated every image-related task. Is the Viola-Jones face detector still considered state-of-the-art, or have CNNs surpassed its performance?


texts:
Can someone explain me the logic behind the confusion matrix?

True Positive (TP): prediction is POSITIVE, actual outcome is POSITIVE, result is 'True Positive' - No questions.
False Negative (FN): prediction is NEGATIVE, actual outcome is POSITIVE, result is 'False Negative' - Why is that? Shouldn't it be 'False Positive'?
False Positive (FP): prediction is POSITIVE, actual outcome is NEGATIVE, result is 'False Positive' - Why is that? Shouldn't it be 'True Negative'?
True Negative (TN): prediction is NEGATIVE, actual outcome is NEGATIVE, result is 'True Negative' - Why is that? Shouldn't it be 'False Negative'?


texts: Tests to find which theory agrees better with observation

texts:
I have three curves ( 1.> observation: yobs , 2.> theory-1: yth1 , 3.> theory-2: yth2 ). All of these curves are functions of a single variable (say variable x.) From a computational perspective, all these curves can be thought of as arrays with discrete values. To get an idea of error bars around  yobs, I use 100 numerical simulations. All arrays, yobs, yth1, yth2 and all 100 simulations have the same length (same number of values).

What statistical test can tell me which one of these two theories (yth1 or yth2)  is a more accurate representation of the observation ( yobs ) inside a given distance range ([x1, x2])?

Below is a schematic diagram of yobs, yth1 and yth2. The orange shaded region around yobs shows  error bar gotten from 100 simulations.

I have already tried  analysis using the formulae given underneath (here,  denotes covariance matrix obtained from the 100 simulations). But, I get very high values of / degree of freedom for comparisons of yth1 with yobs , and yth2 with yobs .  As such, I am looking for tests other than the  test.

Another test that I have tried is the method of fractional errors (shown in the two formulae given below). While the fractional tests tell me which theory is a more accurate model for observations, it does not use errors from 100 simulations. So, I am also trying to find an alternative to this.


texts: Why use Variational Autoencoders VAE instead of Autoencoders AE in Anomaly Detection?

texts:
I have read many papers that recommend using Variational Autoencoders over Autoencoders since they have a more probabilistic approach and the ability to use KL divergence on the latent dimension. But when trying to test both networks I find that the variability of the output in  Variational Autoencoders is reducing the accuracy of the network and I am getting better results when using Autoencoders. I am still working on very simple data and training my network on normal images that do not have any augmentation or changing background.

Does the performance of Variational Autoencoders increase with harder data or is there any other reason to choose it over Autoencoders?
Or do Autoencoders perform better in anomaly detection?


texts: How to retrieve results summary from statsmodels GLM with regularization?

texts:
I'm trying to fit a GLM to predict continuous variables between 0 and 1 with statsmodels. Because I have more features than data, I need to regularize. statsmodels has very few examples, so I'm not sure if I'm doing this correctly.
import statsmodels.api as sm

logistic_regression_model = sm.GLM(
    y,  # shape (num data,)
    X,  # shape (num data, num features)
    link=sm.genmod.families.links.logit)
results = logistic_regression_model.fit_regularized(alpha=1.)
results.summary()

When I run this, asking for a summary raises an error.
NotImplementedError                       Traceback (most recent call last)
<ipython-input-167-169b134cd8cb> in <module>
      7     link=sm.genmod.families.links.logit)
      8 results = logistic_regression_model.fit_regularized(alpha=1.)
----> 9 results.summary()

/opt/conda/lib/python3.6/site-packages/statsmodels/base/model.py in summary(self)
   1115         Not implemented
   1116         """
-> 1117         raise NotImplementedError
   1118
   1119

NotImplementedError:

How do I get a summary of the fit model?


texts: Dealing with feature vectors of variable length

texts:
How does one deal with a feature vector that can vary in size?

Let's say per object, I calculate 4 features. In order to solve a certain regression problem, I may have 1, 2, or more of these objects (no more than 10).  Thus, the feature vector is 4*N in length. How is this normally addressed?

The objects represent physical objects (e.g. other people) w.r.t. an observer. For a time slice, an object can be placed laterally, longitudinally, have some speed and some heading (4 features). Trying to solve: where should a person feel most comfortable. In some cases there is only 1 object, but there can be 2 or more.

Disclaimer: I have limited knowledge on ML approaches. I had classes in college years ago and took Andrew Ng's ML course online as a refresher but otherwise not up to speed. A starting place to look is appreciated.


texts: I am doing text mining to extract topics from documents. I started with Latent Dirichlet Allocation (LDA), which worked great, but then I came across TF-IDF with K-Means clustering, which worked better for me. I'd like to evaluate both but I can'f find any useful validation or metric to compare these two. How is it possible to compare these two with a useful metric?


texts: What methods exist to compare two corpora for suitability of supervised model use?

texts:
Whereas so many Kaggle contests I have done have a test set which is withheld from the entire single dataset, often times I find myself asking: if I train a supervised model one this one corpus, how well could the resulting model make predictions on a second corpus?  So this would suppose that the truth labels from the first corpus would be relevant to the second corpus, but for the problem I am working I think it is possible.  We have truth data for the first corpus, but none for the second corpus and probably not the time/manpower to manually annotate the second corpus.

So are there any methodologies that can be performed in practice which give us an understanding if two corpora are closely enough related that this could be possible?


texts: Is there any literature on selection of dimensions for word embeddings?

texts: I'm looking for any papers/articles on how the word embedding dimensions affect the performance in NLP tasks. Specifically, has anyone ever used low dimensional(<10) word embeddings and achieved good results?


texts: Large mean squared error in sklearn regressors

texts:
I'm a beginner in machine learning and I want to build a model to predict the price of houses. I prepared a dataset by crawling a local housing website and it consists 1000 samples and only 4 features (latitude, longitude, area and number of rooms).

I tried RandomForestRegressorand LinearSVR models in sklearn, but I can't train the model properly and the MSE is super high.

MSE almost equals 90,000,000 (the true values of prices' range are between 5,000,000 - 900,000,000)

Here is my code:

import numpy as np
from sklearn.svm import LinearSVR
import pandas as pd
import csv
from sklearn.preprocessing import normalize
from sklearn.model_selection import train_test_split

df = pd.read_csv('dataset.csv', index_col=False)
X = df.drop('price', axis=1)

X_data = X.values
Y_data = df.price.values
X_train, X_test, Y_train, Y_test = train_test_split(X_data, Y_data, test_size=0.2, random_state=5)

rgr = RandomForestRegressor(n_estimators=100)
svr = LinearSVR()

rgr.fit(X_train, Y_train)
svr.fit(X_train, Y_train)

MSEs = cross_val_score(estimator=rgr,
                         X=X_train,
                         y=Y_train,
                         scoring='mean_squared_error',
                         cv=5)

MSEsSVR = cross_val_score(estimator=svr,
                         X=X_train,
                         y=Y_train,
                         scoring='mean_squared_error',
                         cv=5)

MSEs *= -1
RMSEs = np.sqrt(MSEs)

print("Root mean squared error with 95% confidence interval:")
print("{:.3f} (+/- {:.3f})".format(RMSEs.mean(), RMSEs.std()*2))
print("")

Is the problem with my dataset and count of features? How can I build a prediction model with this type of dataset?


texts: I got a dataset made out of tweets and I need to classify them into topics. For topic modelling with LDA I have cleaned out the dataset (removing stopwords, mentions, symbols, etc). Do I need to remove also retweets in order not to influence the LDA? If so, how should I finally classify all tweets since I could not use per-document-per-topic probability?


texts:
I have trained a SVM for image classification using RGB histogram as features and a couple of other ones.

These are my feature and label sizes:

STATUS] feature vector size (11244, 525)
[STATUS] training Labels (11244,)
[STATUS] training labels encoded...
[STATUS] feature vector normalized...
[STATUS] target labels: [0 0 0 ... 1 1 1]
[STATUS] target labels shape: (11244,)

I am using sklearn's train_test_split with ratio of 0.15 for test.

Following is the classification report(100% recall and precision on test). This is weird!!

Tuning hyper-parameters for precision

Best parameters set found on development set:

{'C': 1, 'kernel': 'linear'}

Grid scores on development set:

0.991 (+/-0.005) for {'C': 1, 'gamma': 0.001, 'kernel': 'rbf'}
0.852 (+/-0.003) for {'C': 1, 'gamma': 0.0001, 'kernel': 'rbf'}
0.993 (+/-0.003) for {'C': 10, 'gamma': 0.001, 'kernel': 'rbf'}
0.991 (+/-0.005) for {'C': 10, 'gamma': 0.0001, 'kernel': 'rbf'}
0.999 (+/-0.002) for {'C': 100, 'gamma': 0.001, 'kernel': 'rbf'}
0.993 (+/-0.003) for {'C': 100, 'gamma': 0.0001, 'kernel': 'rbf'}
1.000 (+/-0.000) for {'C': 1000, 'gamma': 0.001, 'kernel': 'rbf'}
0.999 (+/-0.002) for {'C': 1000, 'gamma': 0.0001, 'kernel': 'rbf'}
1.000 (+/-0.000) for {'C': 1, 'kernel': 'linear'}
1.000 (+/-0.000) for {'C': 10, 'kernel': 'linear'}
1.000 (+/-0.000) for {'C': 100, 'kernel': 'linear'}
1.000 (+/-0.000) for {'C': 1000, 'kernel': 'linear'}

Detailed classification report:

The model is trained on the full development set.
The scores are computed on the full evaluation set.

precision recall f1-score support

0 1.00 1.00 1.00 574
1 1.00 1.00 1.00 1113

micro avg 1.00 1.00 1.00 1687
macro avg 1.00 1.00 1.00 1687
weighted avg 1.00 1.00 1.00 1687

# Tuning hyper-parameters for recall

Best parameters set found on development set:

{'C': 1, 'kernel': 'linear'}

Grid scores on development set:

0.986 (+/-0.006) for {'C': 1, 'gamma': 0.001, 'kernel': 'rbf'}
0.633 (+/-0.009) for {'C': 1, 'gamma': 0.0001, 'kernel': 'rbf'}
0.988 (+/-0.005) for {'C': 10, 'gamma': 0.001, 'kernel': 'rbf'}
0.986 (+/-0.006) for {'C': 10, 'gamma': 0.0001, 'kernel': 'rbf'}
0.998 (+/-0.003) for {'C': 100, 'gamma': 0.001, 'kernel': 'rbf'}
0.988 (+/-0.005) for {'C': 100, 'gamma': 0.0001, 'kernel': 'rbf'}
1.000 (+/-0.001) for {'C': 1000, 'gamma': 0.001, 'kernel': 'rbf'}
0.998 (+/-0.003) for {'C': 1000, 'gamma': 0.0001, 'kernel': 'rbf'}
1.000 (+/-0.000) for {'C': 1, 'kernel': 'linear'}
1.000 (+/-0.001) for {'C': 10, 'kernel': 'linear'}
1.000 (+/-0.001) for {'C': 100, 'kernel': 'linear'}
1.000 (+/-0.001) for {'C': 1000, 'kernel': 'linear'}

Detailed classification report:

The model is trained on the full development set.
The scores are computed on the full evaluation set.

precision recall f1-score support

0 1.00 1.00 1.00 574
1 1.00 1.00 1.00 1113

micro avg 1.00 1.00 1.00 1687
macro avg 1.00 1.00 1.00 1687
weighted avg 1.00 1.00 1.00 1687

BUT, on new images(sampled from same distribution), I am not getting even a single classification right. What could be going wrong?


texts: Does PIFUHD 3D reconstruction model preserve sizes?

texts:
This is little conceptual question. I am working on 3D reconstruction using deep learning from 2D images. I came across PIFUHD model which is developed by Facebook and I have a question regarding that does this model preserve sizes of humans like arm length head diameter or any other body part? If it does can you please cite any source where you came across. Or do you know any other pre-trained model which does that? I would prefer deep learning based approaches but even if you have any other ways please mention here any help is appreciated.
For more detailed idea about what I am trying to say: Suppose there is a person in 2D image is 5 feet 11 inches does 3D reconstructed image will give me 5 feet 11 inches tall person or there is fixed height person is returned back or there is no fixed such things?
Any help is appreciated.


texts: I have to find homography transform parameters which are 8 in number with the help on CNN

texts: I have to find homography transform parameters which are 8 in number..I m using a pretrained residual network. The problem I am facing is I am not able to use model.fit() as it takes argument such as x-train and labels and I don't have any labels. In dataset only stable and unstable videos given?


texts: how to use tensorflow graphs in multithread？valueerror：tensor a must be from the same graph as tensor b！

texts:
I am doing instance detect and image retrieval task by Keras and Tensorflow as backend.

I plan to use multi thread to load two model, I load maskrcnn in a thread and load mobile net in another one.

I load the maskrcnn in a thread successfully, but I failed to load mobile net in another thread, and:

  show:ValueError: tensor a must be from the same graph as tensor b.

The code is as below:

Merge.py

from keras import backend as K

g1=tf.Graph()
g2=tf.Graph()
sess1=tf.Session(graph=g1)
sess2=tf.Session(graph=g2)

def intiMaskrcnn():
     with g1.as_default():
           with sess1.as_default():
                   Model1=........

def instanceDetect():
       K.set_session(sess1)
       with g1.as_default():
             Model1.predit()
             ............

def intiMobilenet():
     with g2.as_default():
           with sess2.as_default():
                   Model2=........

def Retrieval():
       K.set_session(sess2)
       with g2.as_default():
             Model2.predit()
             ............

Thread1.py

intiMaskrcnn()
instanceDetect()

Thread2.py

intiMobilenet()
Retrieval()

Where am I going wrong?


texts:
How to implement clipping the reward in DQN in keras? especially how to implement clipping the reward?

Is this pseudo code correct:

if reward<-threshold reward=-1
elseif reward>threshold reward=1
elseif -threshold<reward<threshold reward=reward/threshold

And if reward is always positive how we can change clipping the reward?


texts: Random Forest, Duplicating Data increases Accuracy. Why?

texts: I duplicated my training data for the random forest classifier (Sklearn) and the accuracy of the prediction increased by about 3%. Why?


texts: Time Series prediction using LSTMs: Importance of making time series stationary

texts:
In this link on Stationarity and differencing, it has been mentioned that models like ARIMA require a stationarized time series for forecasting as it's statistical properties like mean, variance, autocorrelation etc are constant over time. Since RNNs have a better capacity to learn non-linear relationships (as per given here: The Promise of Recurrent Neural Networks for Time Series Forecasting) and perform better than traditional time series models when the data is large, it is essential to understand how stationarized data would affect its results. The questions I need to know the answer of are as follows:

In case of traditional time series forecasting models, stationarity in time series data makes it easier to predict, why and how?
While building a time series prediction model using LSTMs, is it important to make the time series data stationary? If so, then why?


texts:
I would like to know what type of computer configuration to use to perform datascience on medium to low quantity of data (< 100 000 matrix line x 50 parameters).

I have read some blog on people building their own system, but I am performing regressions essentially. Sklearn is quite well designed to me and 1 GPU with cuda would be enough for the rest.

I am surprise to read no feedback about "standard" computer designed for gaming. They are maybe missing a bit of RAM, but after that they look not so bad.

For example, a computer like an ASUS ROG HURACAN present interesting configuration for an accessible price.

Am I missing a key point? any feedback to share? benchmark to share?

thx for help!


texts: Pytorch testing/validation accuracy over 100%

texts:
So I was training my CNN for some hours when it reached 99% accuracy (which was a little bit too good, I thought). But then it didn´t stop and it went higher than 100%. So I thought, that must be because I messed something up in the accuracy calculater function, but I didnt find the mistake. Can somebody tell me if something is wrong here?

def accuracy(model, dataset):
    # set .eval() to ignore dropout
    model = model.eval()
    accuracy = 0
    total = 0
    correct = 0
    for _, data in enumerate(dataset):
        samples, targets = data
        samples = samples.float().cuda()
        targets = targets.float().cuda()

        predictions = model(samples)

        total += targets.size(0)
        correct += predictions.eq(targets.data).sum().item()

    accuracy = 100 * correct/total
    return accuracy

Arguments: the model, and the dataset (for example the validation set or the testing set)
Thanks for your help :)


texts: Why does this paper claim to have found a minimal width of $d_{in}+1$?

texts: Why does this paper (click the link) claim to have found a minimal width of $d_{in}+1$ in the abstract? I mean, if you read the main result, it seems like they only find a universal approximator with width $d_{in} + d_{out}$. What am I missing or did they make a mistake? I do not completely understand all the math but I do think I get the gist of it, and also $d_{in}+1$ is not even mentioned when showing a bound of the difference between the function approximated by the neural network and the actual function...


texts:
I have some non-english words/sentences in my data.
I tokenized my text and tried using nltk.corpus.words.words() but its not really helpful as it also removes the brand names, company names, like NLTK etc. I need some solid solution for the purpose.
Here's what I tried:
def removeNonEnglishWordsFunct(x):
    words = set(nltk.corpus.words.words())
    filteredSentence = " ".join(w for w in nltk.wordpunct_tokenize(x) \
                                if w.lower() in words or not w.isalpha())
    return filteredSentence

string = "NLTK testing man Apple Confiz Burj Al Arab Copacabana Palace Ｗは比較的新しくてきれいなのですが Ｓｈｅｒａｔｏｎ hotelは時々 ＮＹらしい小さくて清潔感のない部屋"

res = removeNonEnglishWordsFunct(string)
Output: testing man Apple Al Palace

Expected output: NLTK testing man Apple Confiz Burj Al Arab Copacabana Palace Ｓｈｅｒａｔｏｎ hotel
```


texts: How do get the first 15 min high from OHLC data through pandas?

texts:
Here is a Dataframe which has OHLC minute wise data from 2011-2021
I want to make another column named "first15 high" where I want to have first 15 minute high i.e. 9:15 to 9:30 highest high for that day.
The desired output (in the yellow column )is below. The Dataframe has more than 10 years of data (i.e. contains more than 2000 days).