texts: Data Mining - Intent matching and classification of text

texts:
PROBLEM

Suppose you have a list of 100,000+ google queries related to travel bookings. For example:

hotels in london
barcelona flight
city breaks to berlin
khao san road hostel
luxury holiday to paris
new york business class flight price
disneyland trip...

How can I extract the location i.e., London
Classify the Line of Business i.e., flight, hotel, package, etc
Classify the affinity i.e., luxury, family, city break, beach etc
Use this info to record the frequency of various pattern that exists in the keywords

i.e. **keyword pattern**                **frequency**
(destination) hotel                          xxx
flight to (destination)                      yyy
(theme) (destination) hotel                  zzz

POTENTIAL SOLUTIONS

Manual - get an exhaustive as possible list of locations (most of
the queries will be for tourist destinations) and look for a match
against the keywords. Again, compare the keywords against possible
Line of Business Identifiers and Affinity Identifiers
Google Cloud Natural Language API - This can be used to analyse
Entities and Sentiment of text. e.g.
     hotels in london -> entity(hotels), entity(london)
     barcelona flight -> entity(flight), entity(barcelona)
This isn't very powerful and only supports english.
Machine learning - seems difficult as I haven't got any descriptors
for the keywords. Would Naive Bayes be applicable or SVM?

I'd preferably like to run any solution in R too.

Could someone please suggest a direction/potential solution?


texts: One hot vector output in classification task

texts:
I'm working on CNN model and I used one hot vector type of labels. The number of classes is 3: [1,0,0], [0,1,0], [0,0,1].
net(x)

I'm getting such an output: [0.8439, 0.1355, 0.0757], which is obviously 1st class. The question: why a sum of values in this vector exceeds 1? Also, I got earlier even one negative value of those 3. On what it is depending and how to know what these "outputs" could be.


texts: R: lattice equivalent of density2d in ggplot?

texts:
What would be the equivalent of geom_density2d in lattice? In essence I'm trying to create this graph with lattice:

I don't think contourplot or levelplot is what i want and when trying it, it gives me a blank plot?


texts: .ValueError: Found input variables with inconsistent numbers of samples: [1388, 2083]

texts:
I'm typing below code

import numpy as np
from sklearn.model_selection import KFold

X = churn[["gender code","senior_citizen","partner code","tenure","dependents code","phone_service code","multiple_lines code",
           "online_backup code","device_protection code","tech_support code","streaming_tv code","streaming_movies code",
           "paperless_billing code", "payment_method_Bank transfer (automatic)", "payment_method_Electronic check",
           "payment_method_Credit card (automatic)", "payment_method_Mailed check","internet_service_DSL",
           "online_security code","internet_service_Fiber optic","internet_service_No","contract_Month-to-month",
           "contract_One year","contract_Two year","tenure","monthly_charges", "total_charges"]]
y = churn["churn"]

kf = KFold(n_splits=5)
kf.get_n_splits(X)

print(kf)

for train_index, test_index in kf.split(X):
    print("TRAIN:", train_index, "TEST:", test_index)
    X_train, X_test = X.loc[train_index], X.loc[test_index]
    y_train, y_test = y.loc[train_index], y.loc[test_index]

# Print Confusion Matrix
print("***** Confusion Matrix *****")
cm_labels = churn["churn"].unique()
print(cm_labels)
print(confusion_matrix(y_test, y_pred, labels=cm_labels))

# Print Report
print()
print("***** Report *****")
print(classification_report(y_test, y_pred))

# Check F1 value
print()
print("***** F1 *****")
f1 = f1_score(y_test, y_pred, average='weighted')
print ("F1 = ", f1)

And the output and the error show like this

KFold(n_splits=5, random_state=None, shuffle=False)
TRAIN: [1389 1390 1391 ... 6940 6941 6942] TEST: [   0    1    2 ... 1386 1387 1388]
TRAIN: [   0    1    2 ... 6940 6941 6942] TEST: [1389 1390 1391 ... 2775 2776 2777]
TRAIN: [   0    1    2 ... 6940 6941 6942] TEST: [2778 2779 2780 ... 4164 4165 4166]
TRAIN: [   0    1    2 ... 6940 6941 6942] TEST: [4167 4168 4169 ... 5552 5553 5554]
TRAIN: [   0    1    2 ... 5552 5553 5554] TEST: [5555 5556 5557 ... 6940 6941 6942]
***** Confusion Matrix *****
['No' 'Yes']
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-95-d28075d9c879> in <module>
     24 cm_labels = churn["churn"].unique()
     25 print(cm_labels)
---> 26 print(confusion_matrix(y_test, y_pred, labels=cm_labels))
     27
     28 # Print Report

~\Anaconda333\lib\site-packages\sklearn\metrics\_classification.py in confusion_matrix(y_true, y_pred, labels, sample_weight, normalize)
    266
    267     """
--> 268     y_type, y_true, y_pred = _check_targets(y_true, y_pred)
    269     if y_type not in ("binary", "multiclass"):
    270         raise ValueError("%s is not supported" % y_type)

~\Anaconda333\lib\site-packages\sklearn\metrics\_classification.py in _check_targets(y_true, y_pred)
     78     y_pred : array or indicator matrix
     79     """
---> 80     check_consistent_length(y_true, y_pred)
     81     type_true = type_of_target(y_true)
     82     type_pred = type_of_target(y_pred)

~\Anaconda333\lib\site-packages\sklearn\utils\validation.py in check_consistent_length(*arrays)
    210     if len(uniques) > 1:
    211         raise ValueError("Found input variables with inconsistent numbers of"
--> 212                          " samples: %r" % [int(l) for l in lengths])
    213
    214

ValueError: Found input variables with inconsistent numbers of samples: [1388, 2083]
```


texts: Feature-to-parameter mapping in neural networks

texts:
For neural networks, can we tell which parameters are responsible for which features?

For example, in an image classification task, each pixel of an image is a feature.
Can I somehow find out which parameters encode the learned information, say from the top-left pixel of my training instances?


texts: Which of the 180 algorithms in R's caret package are feasible?

texts:
R's caret package works with 180 models. The author warns that some of the package may be intractably slow or less accurate than top-choice models.

The author is not wrong about this. I've tried to train Boruta and evtree models and had to give up after they ran > 5 hours on a cluster.

The author links to a set of machine learning benchmarks, but those only cover performance of a small number of algorithms, comparing different implementations.

Is there some other resource I can turn to, for guidance on which of the 180 models are worth trying, and which will be very inaccurate or unreasonably slow?


texts: NLP based Data Preprocessing Method to Improve Disease Name Prediction Using CRF and Word Embedding

texts:
I built a model( using CRF along bi lstm) to Predict New Disease Name/Entities from medical text data but the problem is Disease name appears only 5,6 times in 1 text file  but on average text file consists of 500-1000 words and this increase False negative (FN).  when i test model on newly/real time Data set its only Predict 60% disease Names (On new dataset).

In total i have 16000 words having  (O-tags which represent Irrelevant words) and only 220 words having B-Disease tags.
My model precision is 73 % and recall 61% f1 score is 66 %. Parameter on which i trained model

params = {
    'dim': 300,
    'dim_chars': 100,
    'dropout': 0.5,
    'num_oov_buckets': 1,
    'epochs': 75,
    'batch_size': 7,
    'buffer': 15000,
    'char_lstm_size': 25,
    'lstm_size': 100,
    'words': str(Path(DATADIR, 'vocab.words.txt')),
    'chars': str(Path(DATADIR, 'vocab.chars.txt')),
    'tags': str(Path(DATADIR, 'vocab.tags.txt')),
    'glove': str(Path(DATADIR, 'glove.npz'))
}
[file having complete code][1]

Case #2 : When i trained model only on those lines which include disease Names this cause High False Positive  .

Q1 .Should i remove half lines from every text file so irrevlant dataset decrease.

Q2. Or increase data set without removing half lines from a text file (increase train data set size rather using 80 files  train data on 200 files) As i am using Neural Architectures for Named Entity Recognition and Neural Architectures Want more training data Set

I am using following Code / Library , tf_ner

i am using 2nd model mention in the library  "chars_lstm_lstm_crf"

This model include :

 GloVe 840B vectors

 Chars embeddings

 Chars bi-LSTM

 Bi-LSTM

 CRF

Related Research Paper
Neural Architectures for Named Entity Recognition by Lample et al.
==>>>

Research Paper


texts: Is the multilayer perceptron only able to accept 1d vector of input data? If yes, why is this so?

texts:
I am going through the tutorial at the link below which uses MNIST handwritten digit database.

https://machinelearningmastery.com/handwritten-digit-recognition-using-convolutional-neural-networks-python-keras/

The 28x28 sized image data has to be reshaped into a 1D vector of 784 pixels. 28x28=784. Why does the multilayer perceptron insist on only a 1D vector of input data? There are no problems with convolutional and recurrent neural networks accepting input shape of higher dimensions.


texts: No correlation found between dataset features

texts:
I'm trying to build a classification model that predicts the price of New York taxi trips (year 2018).
Datasource page

Since the original file is very large (112 234 626 rows), I constructed the smaller sample file (1 000 000 rows) to make the first exploratory analysis.
By using this sample file I constructed Heatmap chart to display the correlations between dataset features.
I found out that the model's dependent variable (fare_amount) is only correlated to trip_distance variable. The other independant variables didn't have any considerable correlation.

Does this mean that the dataset I use is not appropriate for the classification task?
Or I haven't solved the task correctly?
How many independent variables should correlate to the dependent variable to count it "good enough" for further analysis?
Which method should I prefer for feature selection and dimensionality detection? Should it be Heatmap chart, PCA or something else?

My code in GitHub


texts: Is there a python library for reformatting names?

texts:
I have a list of several hundred thousand electrical assets named in multiple databases which I am trying to reformat to fit into a universal naming convention.

I know I could solve this problem hard-coding and using regex, but I wanted to try to solve this using machine learning in python and wanted to know if anyone could recommend me a library or method to try tackling this task.

The idea is that there are a bunch of devices for clients we serve which are indexed in multiple databases with different names and I'd like to rename them all to have the same name for all current and future devices added in.

For the most part, it should be relatively easy for the human eye to match up the different names and it is not complicated to write a function that does the same, but as I mentioned I think it would be fun to try to do this as an NLP task:

Example of my training data of before -> after:

MALIBU BEACH 4KV CIRCUIT BREAKER -> MALIBU BEACH 4KV CB
MALIBU BEACH CIRCUIT BREAKER -> MALIBU BEACH 4KV CB
MALIBU BEACH CB -> MALIBU BEACH 4KV CB
MALIBU BCH CB -> MALIBU BEACH 4KV CB
BEVERLY HILLS 3N -> BEVERLY HILLS NO.3N BANK
BEVERLY HILLS BK 3N -> BEVERLY HILLS NO.3N BANK
BEVERLY HILLS NO.3N BK -> BEVERLY HILLS NO.3N BANK
BEVERLY H BANK 3N -> BEVERLY HILLS NO.3N BANK

From the perspective of writing a function to do this, I'd use regex to capture things like the voltage level, the device type, and city name and then reformat it to the fixed format.

From an NLP perspective, if I create a bunch of before -> after cases to train a model, would it be able to handle things like the abbreviations?

Such as figuring out on its own that MALIBU BCH is a reference to MALIBU BEACH like a human would, and apply the logic of matching abbreviations in training to future information passing through

Or do I need to create a separate dictionary for the model to reference something like a list of possible abbreviations?

Would a library like NLTK be the right choice for handling a problem such as this?

The upside I see to this method is that by building a ML solution, I would lay out the framework to have a more robust tool to fix other similar non-uniform data issues on the system rather than just this specific one by going the hard-coding method.


texts: How to add a calculated column in a Pandas dataframe?

texts:
I am new to Python/Pandas so I'm struggling a bit here.

I have a dataframe with air quality data from 2016 to 2020. I want to calculate the annual rate of change for each measured value to compare them with the value the year before at the same day and month.

These are the first lines of the dataframe.

         Date Country       City Specie count   min   max median variance
0  2020-02-23      CR   San José   pm25    20  13.0  53.0   25.0  1232.00
1  2020-04-04      CR   San José   pm25    23  17.0  57.0   38.0  1302.57
2  2020-04-24      CR   San José   pm25    23  30.0  80.0   59.0  1966.13
3  2020-01-14      CR   San José   pm25    24  13.0  34.0   21.0   379.55
4  2020-02-07      CR   San José   pm25    23  57.0  95.0   72.0   838.97

Does anybody have an idea as to how I can proceed?


texts: Classification vs Regression Model what should I choose?

texts: I am working on a problem like 'customers next month revenue prediction'. Here revenue will be the target variable. Again we actually segment the customers based on there revenue(like if they give less than 200 they will be in category 'A' else 'B'). I have to predict both(provable revenue + category). What will be the right approach, choose a regression model and predict the revenue and then categorized or i should choose separate model(regression for revenue, classification for category).


texts: Not enough memory for operations with Pandas

texts:
Wes McKinney, the author of Pandas, writes in his blog that
"... my rule of thumb for pandas is that you should have 5 to 10 times as much RAM as the size of your dataset. So if you have a 10 GB dataset, you should really have about 64, preferably 128 GB of RAM if you want to avoid memory management problems."
I frequently use Pandas with datasets not much smaller than my RAM (16GB). So I wonder, what are   some practical implications of these "memory management problems"? Could anyone provide more insights into this? Does it mean it will store data in virtual memory on disk and therefore be very slow?


texts: Merge and Un-merge multiple models in Keras

texts:
I have some sale data for various products (P1, P2, and P3) in different cities (A, B, C). I wish to develop a model for each product to predict their sales. Since I only have data for total sales in each city, I want to build a network that merges three sub-networks at the output layer. After training, I want to un-merge my model and obtain an individual model for each product. Based on the questions here, my code is like:

from keras.models import Sequential, Model
from keras import layers
from keras import Input
# model for P1
input_p1 = Input(shape=(64,))
p1 = layers.Dense(32, activation='relu')(input_p1)
p1 = layers.Dense(32, activation='relu')(p1)
p1 = layers.Dense(10, activation='softmax')(p1)
p1_sale = layers.Dense(1)(p1)

# model for P2
input_p2 = Input(shape=(64,))
p2 = layers.Dense(32, activation='relu')(input_p2)
p2 = layers.Dense(32, activation='relu')(p2)
p2 = layers.Dense(10, activation='softmax')(p2)
p2_sale = layers.Dense(1)(p2)

# model for P3
input_p3 = Input(shape=(64,))
p3 = layers.Dense(32, activation='relu')(input_p3)
p3 = layers.Dense(32, activation='relu')(p3)
p3 = layers.Dense(10, activation='softmax')(p3)
p3_sale = layers.Dense(1)(p3)

added = keras.layers.Add()([p1_sale, p2_sale, p3_sale])
model = keras.models.Model(inputs=[input_p1, input_p2, input_p3], outputs=added)

My question is how can we split the added model into three separate models for P1, P2, and P3 after I train it?


texts: How to convert Scikit Learn logistic regression model to TensorFlow

texts:
I would like to use existing Scikit Learn LogisticRegression model in the BigQuery ML. However, BQ ML currently has a hard limit of 50 unique labels and my model needs to handle more than that.

BQ accepts TensorFlow models, which do not seem to have this limit.

How can I convert existing Scikit logistic regression model to TensorFlow model?


texts: How to address overlap in categories for image classification?

texts:
I am trying to build an image classification model using CNN for classifying food products on the basis of allergens that they contain (i.e. on the basis of food being lactose rich, histamine rich, caffeine-rich, etc). But there are overlaps in these categories. For eg, yogurt is both a lactose rich food and histamine rich food. While training, I will have images of yogurt in both the lactose category and histamine category. And if the model is fed with a picture of yogurt, I wish the model to tell me all the categories under which the picture is present(in this case, lactose and histamine rich food). There are a few more food products that are present in multiple categories. I want the model to recognize the food product and tell what all categories the picture is a part of!
How to achieve the same?
Is there another method that would be beneficial?
How to measure accuracy (or any other metric) in such an example?


texts: Complex HTMLs Data Extraction with Python

texts:
Does anybody know a way of extracting data with python from more convoluted website structures? For example, I'm trying to extract data from the players in the ATP profiles, but it's just so complicated I quit. I think they're pulling data from some database in the script and I suspect that even if I tried I wouldn't be able to get it.

I then started using a specialized software called ParseHub, which pulls the data somewhat visually. It's a pretty good software, but they make it slow on purpose just so you buy it, and it is particularly not cheap.


texts: How to interpret fit from regression (decision) tree which has used 0 variables

texts:
I have fit a regression tree to my dataset and the output from summary(tree1) is as follows:

Regression tree:
tree(formula = y ~ X)
Variables actually used in tree construction:
character(0)
Number of terminal nodes:  1
Residual mean deviance:  0.985 = 71680 / 72770
Distribution of residuals:
   Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
-1.0050 -1.0050 -0.3122  0.0000  0.7842  2.2140

So seemingly the tree has found no variables/splits worth including. The reason this is curious is because other methods have found trends in the data. For example a linear regression line found several significant variables and p-value for F-statistic was ~0 (although this is a very noisy dataset based on human behaviour and R-squared was not much bigger than 0).

How can I interpret this regression tree output? There are clearly trends in the dataset as shown by linear regression. Are regression trees more focused on accurate predictions and so won't fit well to a noisy dataset?


texts: Parameterization regression of rotation angle

texts:
Let's say I have a top-down picture of an arrow, and I want to predict the angle this arrow makes. This would be between $0$ and $360$ degrees, or between $0$ and $2\pi$. The problem is that this target is circular, $0$ and $360$ degrees are exactly the same which is an invariance I would like to incorporate in my target, which should help generalization significantly (this is my assumption). The problem is that I don't see a clean way of solving this, are there any papers that try to tackle this problem (or similar ones)? I do have some ideas with their potential downsides:

Use a sigmoid or tanh activation, scale it to the ($0, 2\pi)$ range and incorporate the circular property in the loss function. I think this will fail fairly hard, because if it's on the border (worst prediction) only a tiny bit of noise will push the weights to go one way or the other. Also, values closer to the border of $0$ and $2\pi$ will be more difficult to reach because the absolute pre-activation value will need to be close to infinite.
Regress to two values, a $x$ and $y$ value and calculate the loss based on the angle these two values make. I think this one has more potential but the norm of this vector is unbounded, which could lead to numeric instability and could lead to blow ups or going to 0 during training. This could potentially be solved by using some weird regularizer to prevent this norm from going too far away from 1.

Other options would be doing something with sine and cosine functions but I feel like the fact that multiple pre-activations map to the same output will also make optimization and generalizations very difficult.


texts: Pig Rank function not generating rank in output

texts:
I am facing this bizarre issue while using Apache Pig rank utility. I am executing the following code:

email_id_ranked = rank email_id;
store email_id_ranked into '/tmp/';

So, basically I am trying to get the following result

1,email1
2,email2
3,email3
...

Issue is sometime pig dumps the above result but sometimes it dumps only the emails without the rank. Also when I dump the data on screen using dump function pig returns both the columns. I don't know where the issue is. Kindly advice.

Please let me know if you need any more information. Thanks in advance.

Pig version: Apache Pig version 0.11.0-cdh4.6.0


texts:
I want to prove that my proposed machine learning algorithm (prop_ml) is better than other baseline algorithms (ml_1, ml_2, ml_3) when given a small number of data for training. What I've done is to split a dataset into train and test sets. Then, I've randomly selected small k samples (10, 20, 30, ... 100) from the train set and used them to train the classifiers and used the test set for testing. I've replicated this 5 times to make sure I got some reliable results.

Now, I want to evaluate the results. Any suggestions on a statistical test that I can use to prove that the proposed ml is better or not? Thanks.


texts: DBMS or Software for privacy sensitive data

texts:
We have a dataset of very privacy sensitive people data and want to build a database with it. The data protection department in our company doesn't like the idea that the data scientists are able to see any data specific to a person (even if anonymized). We can't preaggregate the data in the database because there are hundreds of different possible aggregations that could be interesting.

Is there a software or DBMS that could ensure that users can only query aggregated results that contain at least groups of N people?

How else would you solve this problem technically?


texts: Prediction in the training sample with randomforest in r

texts:
I'm using a Random Forest algorithm in order to construct a classification model, and I HAVE to check the accuracy of my rf model in the training sample, but as you can see in this answers :

https://stats.stackexchange.com/a/112052/90446

https://stats.stackexchange.com/a/66546/90446

you can't evaluate the accuracy considering the training samples like this:

predict(model, data=train)

I'm not confortable with the idea of use OOB to get accuracy of the training sample, because the OOB was not used to build the model, how could this be right? I don't know what should I do to predict the fit in the training sample and get the accuracy of the training sample, is it possible or make any sense? When a check the AUC of the prediction of my training sample I get something near of 0.98, but the AUC of the test sample is about 0.7. Is this  due to the limitations of prediction at the training sample or due to Overfitting?


texts: As BERT is bidirectional (uses bi-directional transformer), is it possible to use it for the next-word-predict task? If yes, what needs to be tweaked?


texts: Confusion with initialising weights in a neural network

texts:
Here is some code that initialises weights for an RNN:

u = np.random.randn(hidden_size, input_size) * alpha
v = np.random.randn(hidden_size, hidden_size) * alpha
w = np.random.randn(output_size, hidden_size) * alpha

I have some questions about the above. First, I understand that random.randn generates numbers based around the standard normal distribution. How does doing so contribute to efficiency, and why is it favourable? Similarly, why are the weights then multiplied by the learning rate? Finally where I am most confused, why are the parameters for u hidden_size then input_size, rather than the other way around (same for w)? I understand that changing this order messes with the matrix broadcasting further down the line. But there a strategy behind this particular order?


texts: What are the effects of clipping the reward in stability?

texts:
I am looking for stabilizing my results of DQN, I found clipping is one technique to do it but I did not understand it completely!

1- what are the effects of clipping the reward, clipping the gradient, clipping the error in stability and how makes results more stable?

2- In DQN nature it has written they clipping the reward? Would you please explain this more?

3- which of them are more effective in stability?


texts: How to teach neural network a policy for a board game using reinforcement learning?

texts:
I need to use reinforcement learning to teach a neural net a policy for a board game. I chose Q-learining as the specific alghoritm.

I'd like a neural net to have the following structure:

layer - rows * cols + 1 neurons - input - values of consecutive fields on the board (0 for empty, 1 or 2 representing a player), action (natural number) in that state
layer - (??) neurons - hidden
layer - 1 neuron - output - value of action in given state (float)

My first idea was to begin with creating a map of states, actions and values, and then try to teach neural network. If the teaching process would not succeed I could increase the number of neurons and start over.

However, I quickly ran into performance problems. Firstly, I needed to switch from simple in-memory Python dict to a database (not enough RAM). Now the database seems to be a bottleneck (simply speaking there are so many possible states that the retrieval of actions' values is taking a noticeable amount of time). Calculations would take weeks.

I guess it would be possible to teach neural network on the fly, without the layer of a map in the middle. But how would I choose a right number of neurons on the hidden layer? How could I figure out that I'm loosing periously saved (learned) data?


texts: How to choose learning models for different levels of a Stacking Ensemble?

texts: I was wondering if there is any specific rule for designing the architecture of a Stacking Ensemble (e.g. number of levels, models in each level, and models in general). Also, can there be an overlap between the models of different levels? Thanks!


texts: What is valued more in the data science job market, statistical analysis or data processing?

texts:
Not sure if this question is OT here.  If it is, perhaps move it to meta?  If not:

I'm trained in classical statistics, work in research, and have learned much (and taught) machine learning.  I've got a doctorate, but not in statistics or computer science.  I'm not a software engineer and am only really fluent in R.  Maybe I know a bit of C++ and can write a SQL query, but I wouldn't be the guy you'd hire to set up a data pipeline.

There are plenty of engineers who know maybe a thing or two about statistics and machine learning, but who can build apps in Java, manage a spark cluster, set up a distributed database, etc.  But they wouldn't be the person you'd hire to do any sort of analysis that isn't already canned in python.

Browsing jobs in the private sector, I see a lot of demand for the second type of data scientist.  Is there much demand for the first type?  What is the pay differential in the private sector between the two types?  Did you start as the first type and become more like the second type "on the job?"


texts: How to compare the output of Self-organizing maps?

texts:
I am trying to simultaneously cluster and visualize text documents using Self-organizing maps. Since text documents can be represented in various ways (vector space model, GloVe etc), I am trying to figure out how to tell which representation generates the best map. Measures like Quantization error etc., determine the goodness of the map given a dataset. However, they are not useful for quantitatively telling which representation gives a better output.

Is there a quantitative measure to compare the maps generated using different representations (for example, Tf-idf and GloVe) and tell for which representation the output is better?


texts:
This question is related to "How to grow a list of related words based on initial keywords?"

In the previous question they attempt to get similar words to a given word. However, I am interested in knowing the possibility of doing it to sentences.

As I'm not familiar with this area, my questions are:

Is there a way to do this with sentences (without just considering words)?
What tools are available for it?


texts: Predicting parallel time series with multiple features

texts:
I am trying to predict sales for 2 departmental stores which share similar demographic properties. My goal is to make a single LSTM model to predict sales from these parallel time series having multiple features.

My input features for training would be

+----------+-------+--------------+-------+
|   Date   | Store | DayOfTheWeek | Sales |
+----------+-------+--------------+-------+
| 1/1/2019 | A     |            2 |   100 |
| 1/2/2019 | A     |            3 |   200 |
| 1/3/2019 | A     |            4 |   150 |
| 1/1/2019 | B     |            2 |   300 |
| 1/2/2019 | B     |            3 |   550 |
| 1/3/2019 | B     |            4 |  1000 |
+----------+-------+--------------+-------+

and my output for training would be

+----------+-------+--------------+-------+
|   Date   | Store | DayOfTheWeek | Sales |
+----------+-------+--------------+-------+
| 1/4/2019 | A     |            5 |   220 |
| 1/4/2019 | B     |            5 |   700 |
+----------+-------+--------------+-------+

Problem is that LSTM takes input as 3D i.e (n_sample, n_timesteps, n_features) and I can pass a single time series for a specific store (e.g. A)

If I had a univariate mutiple time series I can reshape my input data as follows and pass it to LSTM.

+----------+---------+---------+
|   Date   | A_Sales | B_Sales |
+----------+---------+---------+
| 1/1/2019 |     100 |     300 |
| 1/2/2019 |     200 |     550 |
| 1/3/2019 |     150 |    1000 |
+----------+---------+---------+

But I need to identify how can I predict parallel multivariate time series? Is there any other way to define in Input LSTM layer that there are 2 time series  with 2 features each i.e (2*2).


texts:
I have an idea for build object detection model and I would like to share it with community in order to see if it makes sense.
Dataset:

I've gathered images with different shapes and annotated them with labeled bounding boxes (I am trying to predict 1 class only).
Images are really large (e.g. 5000x3000), which is why I've trained a CNN Classification model on their PATCHES (299 x 299) for binary classification.

My question is this, is it possible for me to:

Build new neural network but this time train it for BOUNDING BOX REGRESSION on full images (remember that I have annotations (i.e. bbox coordinates))
Use previously trained PATCH CLASSIFIER in order to classify ROI's that were detected by newly trained network for bounding box regression (remember that bounding box coordinates are localizing an object that previous patch classifier was trained to classify).

Is this a "solid" idea for object detection? Or do you guys suggest to fine-tune Faster RCNN or something? Problem is that my data is quite specific (medical data) and I would need to use my own classifier for feature extraction  (not a pre-trained classifier on image net).
Thanks!


texts: How to identify potential customers who are ready to convert in to paid?

texts:
I have data that has actions that perform on my tool and I would like to predict the customers who are ready to convert from free/trail to the paid category.

My data looks like the following:

dummy<-data.frame(license=sample(c("Free","Trail","Paid"),10000,replace = T,prob = c(0.6,0.35,0.05)),
           plan_type=sample(1:5,10000,replace=T),
           action1=sample(0:100,10000,replace = T),
           action2=sample(0:1000,10000,replace = T),
           action3=sample(0:10,10000,replace = T),
           num_days_in_product=sample(0:500,10000,replace = T))

head(dummy)
  license plan_type action1 action2 action3 num_days_in_product
1    Paid         1     100      71       5                 285
2    Free         5      75     438       1                   2
3    Free         1       5     555       7                 389
4    Free         3       4     105       0                 150
5    Free         1      16     348       7                 423
6    Free         5      15     866       8                 270

> table(dummy$license)

 Free  Paid Trail
 6016   516  3468
> prop.table(table(dummy$license))

  Free   Paid  Trail
0.6016 0.0516 0.3468

Let me know if any extra information needed.


texts: How to find optimal sample weights in binary classification/regression problem

texts:
I'm training a model (NN) that gets some data as input and outputs a single value in the range of $[0, 1]$. Right now, the average of the outputs in my dataset is around 0.5, but I know that future data will largely consist of 0.0s, and thus there will eventually be a strong data imbalance towards 0.0. I want the training procedure to be future-proof and scalable, so I'm trying to find a way to automatically rebalance the dataset. My library (Keras) supports sample weights in training, which seems like a straightforward way to do this without losing any information.

Basically, I think what I'm looking for is a function $w(y_i) \rightarrow w_i$ that given a training example $y_i$ gives me a weight $w_i$, so that the weighted average of all training examples $Y$ with the weights $W$ is $0.5$. I know there are many configurations of weights that have this property, but of course the weights should be as close to 1 as possible and definitely $> 0$. I also realize that this is not possible for cases where e.g. all numbers are the same, or all are $< 0.5$. But let's assume my data is diverse enough.

I'm sure I'm not the first one to think of this, but I can't find any solutions / best practices for this case. I guess I could treat it as a little optimization problem of its own, but I hope there is something simpler.


texts: Computing a cumulative distribution function in Python

texts:
I'm trying to compute the distribution function of any of the usual distributions in Python... However, all the methods I've seen involve first drawing N samples from said distribution, and then order them somehow, and then do a cumulative sum.
In Mathematica, I can just do CDF[ChiSquaredDistribution[df],quantile]. If I want another distribution, I just substitute ChiSquaredDistribution for the name of that other distribution.
Is there a simple way, like in Mathematica, to compute a cumulative distribution function in Python?


texts: calculation of average ROC in IMageNet paper?

texts: The IMageNEt paper Image Net. presents the Average ROC curve for the 16 classes in the imagenet data, visit image figure. 8 in the paper. what is the known function to compute this ROC plot. As ROC plot is for a binary classification problem. Is this average ROC plot made in the picture average of ROC's for all the 16 categories. Any Help would be appreciated.


texts: Discovering cross category sales using transactions history (Clustering?)

texts:
I have a set of sales transaction data containing more than 1 item purchased. Every item sold has a category. I would like figure out which categories are most commonly ordered together.

The data is more or less like this:

Transaction ID|Item ID | Sales Quantity|Item Category
1              Apple     1              Fruit
1              Banana    1              Fruit
1              Carrot    2              Vegetable
2              Carrot    1              Vegetable
2              Ice Cream 2              Dessert
3              Squash    2              Vegetable
3              Chocolate 2              Dessert
4              Apple     1              Fruit
4              Carrot    1              Vegetable
4              Doughnut  1              Dessert

Just eyeballing above you can see that there are a high amount of vegetable-dessert pairings on the same transaction. But now imagine that we have 250,000+ transactions in the data set and dozens of categories. I'm looking to discover cross category sales only. Not interested in Apples and Bananas (Fruit-Fruit) pairs.

I think I can teach myself how to code the analysis, but I'm just not sure what this is called or what to Google. Any thoughts?


texts:
I saved my model with this code:
from google.colab import files
torch.save(net, 'model.pth')

# download checkpoint file
files.download('model.pth')

Then uploaded this way and checked on an image (x):
model = torch.load('model.pth')
model.eval()
torch.argmax(model(x))

And on the old session, it worked great, but then I started a new session and tried the code above, and got such an error:
AttributeError: Can't get attribute 'EfficientNet' on <module '__main__'>

Maybe somebody knows how to deal with it?


texts: Is there a closed formula/function for decision trees?

texts: i've been studying gradient boosting so realize the pure algorithm requires a function F/model to get boosted.What is the explicit F on gradient boosting trees?


texts: How to merge columns, value count them and then plot the results?

texts:
How do I get from a dataframe with multiple columns that have similar values and need to be merged:

df1 = pd.DataFrame({'firstcolumn':['ab', 'ca', 'da', 'ta','la'],
'secondcolumn':['ab', 'ca', 'ta', 'da', 'sa'], 'index':[2011,2012,2011,2012,2012]})

To a crosstab that tells me for each year how many values were collected?

Index ab ca da ta sa la
2011 2  0  1  1  0  0
2012 0  2  1  1  1  1

Also, how could then plot the table?


texts: How do I interpret the results of weighted clique percolation if communities are subsuming (not overlapping)?

texts:
I am conducting a network text analysis as part of a systematic literature review of social network analysis in computer-supported learning. My nodes are terms in the included studies, and the ties represent how close these terms are to each other in the text (window size of 10).  This is my first time conducting any sort of network analysis and I am unsure about interpreting my results.

I ran a weighted clique percolation (k=5) on my data in the hopes of finding subcommunities of terms that could give me a quantitative overview of the content of my pool of studies. Instead of finding overlapping subcommunities, I found one large subcommunity subsuming smaller ones (k=5). What can I make out of this data then? Is it a result still work noting?


texts: How to find the best fitting parametric distribution for an empirical dataset (stock returns)?

texts:
Given some real-valued empirical data (time series), I could convert it to a histogram to have an (non-parametric) empirical distribution of the data, but histograms are blocky and jagged.
Instead, I would like to identify the best-fitting parametric distribution from the scipy or scipy.stats libraries of distribution functions, so that I can artificially generate a parametric distribution that closely fits the empirical distribution of my real data and is continuous.
If the empirical data are monthly returns of empirical AAPL stock returns, for example, I know that the parametric Johnson-SU distribution resembles, and can mimic, stock return distributions because of its customizable skew. However, the Johnson SU distribution in scipy requires four input parameters to be calibrated. How can I search for the best parameter settings of this parametric distribution from scipy that fits to the empirical distribution of my sample of AAPL returns?


texts: How to represent linear regression in a decision tree form

texts:
I have read that decision trees can represent any hypothesis and are thus completely expressive. So how do we represent the hypothesis of linear regression in the form of a decision tree ? I am referring to the equation

w0 + w1.x1 + w2.x2 + .... + wn.xn = yn


texts: Is it possible to get the mean coefficent of regression after "Test & Score" in Orange using cross-validation?

texts:
The cross-validation devides the data into n folds and measures the accuracy n times therefore. The displayed RMSE in the Test & Score tool is a mean of all n runs, I think. But is it possible to get the mean coefficents of regression as well?

At the moment my process looks like this:

Loading data from a csv with one target and some features. All are numerical data.
Rank features. Use the most important two features
Use Test&Score with a linear Ridge Regression learner to quantify the regression. I'm using the cross-validation with ten folds in the Test&Score tool.

In this case the data table shows the coefficients independent from the Test & Score tool. Even if I delete the Test & Score thing, there will be no changes on the regression coefficients.


texts: How to Predict the future values of time horizon with Keras?

texts:
I just built this LSTM neural network with Keras

    import numpy as np
    import pandas as pd
    from sklearn import preprocessing
    from keras.layers.core import Dense, Dropout, Activation
    from keras.activations import linear
    from keras.layers.recurrent import LSTM
    from keras.models import Sequential
    from matplotlib import pyplot

    #read and prepare data from datafile
    data_file_name = "DailyDemand.csv"
    data_csv = pd.read_csv(data_file_name, delimiter = ';',header=None, usecols=[1,2,3,4,5])
    yt = data_csv[1:]
    data = yt
    data.columns = ['MoyenneTransactHier', 'MaxTransaction', 'MinTransaction','CountTransaction','Demand']
    # print (data.head(10))
    pd.options.display.float_format = '{:,.0f}'.format
    data = data.dropna ()
    y=data['Demand'].astype(int)
    cols=['MoyenneTransactHier', 'MaxTransaction', 'MinTransaction','CountTransaction']
    x=data[cols].astype(int)

    #scaling data
    scaler_x = preprocessing.MinMaxScaler(feature_range =(-1, 1))
    x = np.array(x).reshape ((len(x),4 ))
    x = scaler_x.fit_transform(x)
    scaler_y = preprocessing.MinMaxScaler(feature_range =(-1, 1))
    y = np.array(y).reshape ((len(y), 1))
    y = scaler_y.fit_transform(y)
    print("longeur de y",len(y))
    # Split train and test data
    train_end = 80
    x_train=x[0: train_end ,]
    x_test=x[train_end +1: ,]
    y_train=y[0: train_end]
    y_test=y[train_end +1:]
    x_train=x_train.reshape(x_train.shape +(1,))
    x_test=x_test.reshape(x_test.shape + (1,))

    print("Data well prepared")
    print ('x_train shape ', x_train.shape)
    print ('y_train', y_train.shape)

    #Design the model - LSTM Network
    seed = 2016
    np.random.seed(seed)
    fit1 = Sequential ()
    fit1.add(LSTM(
        output_dim = 4,
        activation='tanh',
        input_shape =(4, 1)))
    fit1.add(Dense(output_dim =1))
    fit1.add(Activation(linear))
    #rmsprop or sgd
    batchsize = 1
    fit1.compile(loss="mean_squared_error",optimizer="rmsprop")
    #train the model
    fit1.fit(x_train , y_train , batch_size = batchsize, nb_epoch =20, shuffle=True)

    print(fit1.summary ())

    #Model error
    score_train = fit1.evaluate(x_train ,y_train ,batch_size =batchsize)
    score_test = fit1.evaluate(x_test , y_test ,batch_size =batchsize)
    print("in  train  MSE = ",round(score_train,4))
    print("in test  MSE = ",round(score_test ,4))

    #Make prediction
    pred1=fit1.predict(x_test)
    pred1 = scaler_y.inverse_transform(np.array(pred1).reshape ((len(pred1), 1)))
    real_test = scaler_y.inverse_transform(np.array(y_test).reshape ((len(y_test), 1))).astype(int)

    #save prediction
    testData = pd.DataFrame(real_test)
    preddData = pd.DataFrame(pred1)
    dataF = pd.concat([testData,preddData], axis=1)
    dataF.columns =['Real demand','Predicted Demand']
    dataF.to_csv('Demandprediction.csv')

    pyplot.plot(pred1, label='Forecast')
    pyplot.plot(real_test,label='Actual')
    pyplot.legend()
    pyplot.show()

then it generates this result:

After building and training a good model on the historical data, I don't know how I can generate the prediction for future values? For example the demand of the next 10 days. Data are daily.

NB: this is an example of how the data is shaped, the green is the label and the yellow one are the features.
after dropna() (delete null values) it remains 100 data rows, I've used 80 in the training and the 20 in the test.


texts: Is it possible to patent a new general method of optimizing some industrial processes?

texts:
I have finished a new method (implemented as an algorithm/software) which exploring data inputs obtains a sequence of steps, appliable for different industrial processes, that reduces drastically material and human resources as well as energy and time.

For example. This method gives the optimal commands for managing an arm-robot to arrange 7 tones of different size stones in 10 pallets of 1 cubic metre in 60 minutes instead of the current process that needs 3 arm-robots (and others) for performing the same task in 30 pallets and in more than 300 minutes.

Note that for different processes and inputs (a different set of stones) it will produce a different sequence of commands, so it is generating new industrial processes for each new task.

I know that software that implements industrial processes are patentable in US and EU but as I´ve exposed it is not an industrial process but an algorithm to generate some industrial processes.

Is this case of software patentable as a unique-global software for industrial processes or each case must be patented apart?


texts:
I'm trying to build a Convolutional Neural Network model to classify and predict a brain tumor based on images. I am looking for a database containing images of brain tumor.

Can someone introduce some medical databases (small one to test or big ones) with links to download?


texts:
I am working on a breast cancer dataset (http://kdd.org/kdd-cup/view/kdd-cup-2008). I need to perform classification on the data using C4.5 algorithm, after doing any necessary pre-processing.

A section of the report that I have to write is "benchmark models" and I have no idea what that means. I googled the term and it doesn't seem to be something well defined in data mining. Any idea what that means?

Thanks!


texts: LimeTextExplainer for MultiClass classification - Facing issue for explain instance with custom classifier function

texts:
exp = explainer.explain_instance(df_val_final.Description[idx],predproba_list,num_features=5, top_labels=2)

While executing the explain instance of LimeTextExplainer, the above statement keeps on executing continuously with the below warning message. Execution stops only if I interrupt the kernel
C:\ProgramData\Anaconda3\lib\site-packages\fastai\torch_core.py:83: UserWarning: Tensor is int32: upgrading to int64; for better performance use int64 input
  warn('Tensor is int32: upgrading to int64; for better performance use int64 input')
C:\ProgramData\Anaconda3\lib\site-packages\fastai\torch_core.py:83: UserWarning: Tensor is int32: upgrading to int64; for better performance use int64 input
 warn('Tensor is int32: upgrading to int64; for better performance use int64 input')
C:\ProgramData\Anaconda3\lib\site-packages\fastai\torch_core.py:83: UserWarning: Tensor is int32: upgrading to int64; for better performance use int64 input
  warn('Tensor is int32: upgrading to int64; for better performance use int64 input')

I want to use my own custom classifier model and hence I wrote a classifier function - predproba_list, which returns a numpy array of predicted probabilities for the classes
Below is the function code
def predproba_list(test1) :
    pred = learn_clf.predict(test1)
    return np.array(pred[2])

pred[2] vaue is tensor([0.1423, 0.2133, 0.6444]) which i then convert to a numpy array
Can you please advise if the return value of the function is as expected by the explain instance's classifier function, and what could be causing the code to keep on executing without any result?


texts:
I have a table of users' scores like this:

user_id  score duration_of_per_play  start_date
  1        56    313                 2018-01-09
  2        14    560                 2018-08-01
  3        56    313                 2018-01-09
  4        14    560                 2018-08-01

....

now I want to use k-means algorithm the to cluster of users.
just I know that I haven't Missing Values and outliers.

but I don't know:

what is the next step?
do I need to centering data?


texts:
I implemented a spam classifier using Bernoulli Naive Bayes, Logistic Regression, and SVM. Algorithms are trained on the entire Enron spam emails dataset using the Bag-of-words (BoW) approach. Prediction is done on the UCI SMS Spam Collection dataset. I have 3 questions:

During test time, while creating the term-frequency matrix, what if none of the words from my training BoW are found in some of my test emails/smses. Then, wouldn't the document vectors be zero vectors for those datapoints. How should I tackle this?

What if a new word from my test email/sms doesn't exist in BoW?

How do I choose my BoW so as to improve my prediction accuracy?


texts: Which scaler to use for combining different embeddings?

texts:
I am working on an experiment, where I am combining ( average and concatenate ) different embeddings like Elmo, Bert etc, since both embeddings are from different models, I thought, it's better to scale or normalize them with the same scaler and then concatenate or average them. I am confused between different available scalers.
There are around 7 scalers and different normalizer in sklearn :
l1 scaler, l2 scaler, MinMaxScaler , standardscaler etc.
Which scaler and normalization technique would be effective for continuous embedding values before concatenating or averaging them? Any paper, content related to that would be really helpful.


texts: Loss value going down while accuracy remains constant?

texts:
While I am training, it seems like my loss is going down, but my accuracy remains constant throughout training. It always seems to go towards 0.0023 no matter how I tweak my network, input data length, etc.

My input data is a sequence of sample levels before a current sample level. So basically predict the current sample level based on the previous sample levels.

Network is Sequential:

Dense(seq_length)
LSTM(seq_length*10)
Dense(1)

Validation set is 0.33 of the total input samples.

Using Tensorflow Keras on Windows.

CSV of loss, accuracy, val loss, etc:
https://pastebin.com/GPsmeUmg


texts: Question and Answer Chatbot for Customer Support

texts:
I want to build a chatbot that serves as a first line customer support on a  retail website. I have a large log of chat sessions between customers and support professionals that I can use. I am wondering what is the best way to go about building this chatbot.

Here are some ideas I have looked into;

The first thing I did was to breakdown the chat logs into Q&A pairs. Treat each question as a document, compute a term-document matrix and use it to retrieve the most similar question when a customer asks sometihng. The idea was to then simply pick the answer given to the question by the human (with some modifications). However, this gives really abysmal results and does not match to previous questions very well. Even if I get this approach to work well, we would be limited to the questions that already been asked.
Another thing I thought about was to modify something like ALICE and write some AIML for customer support related QA. However, that would take a lot of very precise AIML writint to work well. This solution would not be able to scale to other languages, which is something I want to do.
Another idea I have is to try and cluster the answers given by the customer support staff (they are more likely to be aligned compared to the questions asked by customers). This would give me a sense of what questions are similar. Then I can use something like LSA on the questions and fall back to the first approach.
Yet another way is to build an ontology of the domain specific knowledge and given a question decide which category the question falls into. The questions from the chat logs can then be mapped to the ontology and I could train classifiers for mapping questions to different knowledge areas. Once I reach the leaf nodes, I can give back pre-defined answers. However, my reservation is that mapping chat logs to the ontology would be very tedious. Is there a good way to map the existing QA to the ontology?


texts: How to preprocess tensorflow imdb_review dataset

texts:
I am using tensorflow imdb_review dataset, and I want to preprocess it using Tokenizer and pad_sequences
When I am using the Tokenizer instance and using the following code:
tokenizer=Tokenizer(num_words=100)
tokenizer.fit_on_texts(df['text'])
word_index = tokenizer.word_index
sequences=tokenizer.texts_to_sequences(df['text'])

print(word_index)
print(sequences)

I am getting the error
TypeError: a bytes-like object is required, not 'dict'

What I've tried
store dataset as dataframe and then iterate over the text column, and store it in a list, and then tokenize it.
df = tfds.as_dataframe(ds.take(4), info)
# list to store corpus
corpus = []
for sentences in df['text'].iteritems():
  corpus.append(sentences)

tokenizer=Tokenizer(num_words=100)
tokenizer.fit_on_texts(corpus)
word_index=tokenizer.word_index
print(word_index)

But i'm getting the error
AttributeError: 'tuple' object has no attribute 'lower'

How can I use the text column and preprocess it to feed it to my neural network?


texts:
I am trying to use KNN as an Unsupervised clustering. Yes, I know KNN is supposed to be a used as a classifier, using I was given a task to use it as a clustering model).

I am using this link from sklearn documentation as a reference:

>>> from sklearn.neighbors import NearestNeighbors
>>> import numpy as np
>>> X = np.array([[-1, -1], [-2, -1], [-3, -2], [1, 1], [2, 1], [3, 2]])
>>> nbrs = NearestNeighbors(n_neighbors=2, algorithm='ball_tree').fit(X)
>>> distances, indices = nbrs.kneighbors(X)
>>> indices
array([[0, 1],
       [1, 0],
       [2, 1],
       [3, 4],
       [4, 3],
       [5, 4]]...)
>>> distances
array([[0.        , 1.        ],
       [0.        , 1.        ],
       [0.        , 1.41421356],
       [0.        , 1.        ],
       [0.        , 1.        ],
       [0.        , 1.41421356]])

>>> dist_matrix = nbrs.kneighbors_graph(X).toarray()
array([[1., 1., 0., 0., 0., 0.],
       [1., 1., 0., 0., 0., 0.],
       [0., 1., 1., 0., 0., 0.],
       [0., 0., 0., 1., 1., 0.],
       [0., 0., 0., 1., 1., 0.],
       [0., 0., 0., 0., 1., 1.]])

To help you visualize the problem, this is the data (X):

And this is the desired result:

desired_result = [[0, 1, 2], [3, 4, 5]]

My question is what is a computationally efficient way to construct the underlying clusters from the dist_matrix (or from the indices array)


texts: Compound Annual Growth Rate Function in R Statistics

texts:
I need to calculate the compound annual growth rate for two periods of time: 2014 & 2009. I'm currently working with Excel, I have two columns with data for each year (One column with only data of 2009 and the other only with data of 2014), where each row contains info for 2009 & 2014 for certain category.

I've been trying to calculate the CAGR between the data for each row, but I can't seem to find a function to do precisely that in "R".

I found a package with a growth.rate function for time series, but it doesn't take my data in the order I need (it uses bayes method for velocity of growth, which I don´t need).

Is there any package in "R" where I can get especially that rate? Any tips in how I can accommodate the data so that R reads row per row and calculates like that?


texts:
I am currently learning about Keras and have a problem with the input shape of a dense layer.I am currently trying to the mnist dataset.I understand that the input_shape for the train images is (60000,28,28) also i understand that keras ignores the first dimension as it is the batch size hence the input shape entered in the dense model should be (28,28) but when putting that i get an error yet when i put input shape of(784,) the model runs.Could someone please explain why is that so

(train_images, train_labels), (test_images, test_labels) =
 mnist.load_data()

 print(train_images.shape)

 network = models.Sequential()
 network.add(layers.Dense(512, activation='relu', input_shape=(28,28)))
 network.add(layers.Dense(10, activation='softmax'))


texts:
I am discovering H2O deeplearning and I would like to have your point of view about the performance that's performed my model on classification problem. Do you think my model is overfitting?

dl_fit2 <- h2o.deeplearning(x = predictors, y = response,
                           training_frame = train,
                           validation_frame = valid,
                           epochs = 200,
                           score_validation_samples=10000,
                           score_duty_cycle=0.025,
                           activation = "RectifierWithDropout",
                           hidden = c(80, 10, 80),
                           hidden_dropout_ratios = c(0.2, 0.2, 0.2),
                           loss = "CrossEntropy",
                           rate=0.01,
                           rate_annealing=2e-6,
                           adaptive_rate = FALSE,
                           momentum_start = 0.2,
                           momentum_ramp = 1e7,
                           momentum_stable = 0.4,
                           nesterov_accelerated_gradient = TRUE,
                           l1 = 1e-5,
                           l2 = 1e-5,
                           max_w2=10
)

MSE:  0.009757329
RMSE:  0.09877919
LogLoss:  0.03527449
Mean Per-Class Error:  0.01219048
AUC:  0.9991871
pr_auc:  0.4974259
Gini:  0.9983743

Confusion Matrix (vertical: actual; across: predicted) for F1-optimal threshold:
          NO   YES    Error        Rate
NO     10022   140 0.013777  =140/10162
YES      109 10170 0.010604  =109/10279
Totals 10131 10310 0.012181  =249/20441

Maximum Metrics: Maximum metrics at their respective thresholds
                        metric threshold    value idx
1                       max f1  0.367137 0.987906 212
2                       max f2  0.178219 0.990603 270
3                 max f0point5  0.765904 0.990982 122
4                 max accuracy  0.371864 0.987819 210
5                max precision  0.999979 1.000000   0
6                   max recall  0.001865 1.000000 387
7              max specificity  0.999979 1.000000   0
8             max absolute_mcc  0.367137 0.975640 212
9   max min_per_class_accuracy  0.385816 0.987256 204
10 max mean_per_class_accuracy  0.371864 0.987812 210


texts: How to train a neural network on multiple objectives?

texts:
I have a multi-class neural network classifier that has K classes(products). For every row, only one of the classes will be 1 at a time. Now, this approach works fine if I have only 1 objective to optimize i.e Which of these N products was "clicked" by the user.
But how will I solve this problem if I need to optimize on a 2nd objective i.e Which of these N products was "purchased" by the user?
A purchase event is always preceded by a click event. I can obviously solve this problem by training two separate models - 1 for click and the other for purchase. But purchase data is very low as compared to the click data. And we ran the purchase model on production. It did not perform well.
So how do I take both the click and purchase data and frame my problem as "Which of these N products will be clicked and possibly purchased by the user"? and train a single model.
Any sources or papers in this direction will be really helpful.


texts: Policy-based RL method - how do continuous actions look like?

texts:
I've read several times that Policy-based RL methods can work with continuous action space (move left 5 meters, move right 5.5312 meters), rather than with discrete actions, like Value-based methods (Q-learning)
If Policy-based methods produce probability of taking an action from a current state $S_t$, how can such an action be continuous? We might get 4 probabities for our actions, but we still have to choose a single action from from four:
$A_1:  10\%$
$A_2:  35\%$
$A_3:  5\%$
$A_4:  50\%$
Thus, it's not obvious how can my action be something continuous like:  "turn +19.2345 angles clockwise". Such an action must have already been pre-defined to the "19.2345" value, right?


texts: Clustering by Distributions of Groups of observations


texts: K-means: What are some good ways to choose an efficient set of initial centroids?

texts:
When a random initialization of centroids is used, different runs of K-means produce different total SSEs. And it is crucial in the performance of the algorithm.
What are some effective approaches toward solving this problem? Recent approaches are appreciated.


texts: How to classify named entities of the same type?

texts:
I am doing a project where I am extracting date/time entities from text. I'm using a rule-based system to extract the temporal expressions and ground them to an actual date/time.

The second part of the problem I hope to solve is label the role of each entity discovered. For example, consider the following text: "Leaving at 2pm and back at 4pm". I correctly identified 2pm and 4pm as date/time entities. However, I'm unable to say whether the entity is "start-time", "end-time", or neither.

The question is how do I do this?

I'm new to NLP and ML. Here is an idea I have please tell me if I'm going the right direction:

The plan is to train a logistic regression (or naive bayes?) classifier  using the following features:

The average of the word embedding for each word within a window of the date/time phrase.
The POS tags for each word within a window of the date/time phrase??(Not sure how to pass this in to a logistic regression classifier but just a thought)
The word shapes of the words in the temporal expression??

I'm a little confused as to where to start and would really appreciate some pointers on how to select my features and what classifier would be appropriate.

I'm also open to suggestions on learning resources. There's a lot of NER resources online but not many on how to "role classify" found entities.


texts:
Given a sentence:
"When I open the ?? door it starts heating automatically"
I would like to get the list of possible words in ?? with a probability.
The basic concept used in word2vec model is to "predict" a word given surrounding context.
Once the model is build, what is the right context vectors operation to perform my prediction task on new sentences?
Is it simply a linear sum?
model.most_similar(positive=['When','I','open','the','door','it','starts' ,
                   'heating','automatically'])


texts:
In information retrieval when we calculate the cosine similarity between the query features vector and the document features vector we penalize the unseen words in the query.

Example if we have two documents with features vectors

d1 = [1,1,1,0,0]
d2 = [0,1,1,1,0]

We can see that the two documents have the second feature so if we want to search for the second feature with query vector: q = [0,1,0,0,0] then the cosine similarity between q and d1,d2 will be $1/√3$, and not 1 because that we penalize the other features that we have not mention in the query.

From this discussion I don't understand why penalize it is a good Idea.

Is penalizing unseen features good?
Is there another similarity measure that does not penalize them?


texts: Minimizing overfitting when doing hyperparameter Tuning

texts: Generaly when using Sklearn's GridSearchCV (or RandomizedGridSearchCV), we get best model with best test score even if the model overfits a little bit. How can we compute generalization error efficiently and force GridSearchCV to get best model with minimum generalization error?


texts: Identify credit card shape using machine learning

texts:
I have to approach this task: identify credit card from an image. I am attaching example image below:

I have to identify and localize the credit card from this image. The real challenge is that the card can be placed on any background and the color of the card can change based on the company it belongs to.
In order to solve this task I tried using the tensor flow object detection api. The downside of this api is that it fails to recognize cards which are not in its training data set.
My problem here is I am not concerned with the color of card or what information the card has. I am only concerned about finding the outline of card in an image and isolating the outline of card from rest of the image.
Is there a way using ML/CNN to do this. I tried OpenCV approaches to detect contours but even this approach fails when there is lot of text or other noise in the cards background.


texts: How to select the best parameters for GridSearchCV?

texts:
I've created a couple of models during some assignments and hackathons using algorithms such as Random Forest and XGBoost and used GridSearchCV to find the best combination of parameters. But what I'm not able to understand is how to select those parameters for GridSearchCV. I randomly put the parameters such as
params = {"max_depth" : [5, 7, 10, 15, 20, 25, 30, 40, 50,100],
         "min_samples_leaf" : [5, 10, 15, 20, 40, 50, 100, 200, 500, 1000,10000],
         "criterion": ["gini","entropy"],
         "n_estimators" : [10, 15, 20, 40, 50, 75, 100,1000],
         "max_features" : ["auto", "sqrt","log2"]}

But how do I decide if I could select better parameters which might be computationally better as well? I can't use the same above parameters for a Random Forest Classifier every single time surely?


texts: How to prevent neural network from overfitting on small subset of features

texts:
I'm trying to predict the win probability for a team in a basketball game using a neural network with a single sigmoid output. The input layer consists of a one-hot representation of the players, number of goals(hoops?) for each player, free throws, penalties, etc. For each game I have about 200 training examples each capturing the game state at a particular point in time. My training data spans multiple seasons, so players don't always play for the same team. My hope was that this way I could feed the network a snapshot of the current game state and it would give me the win probability for either team.

Unfortunately, during training the network immediately overfits to the team compositions and memorizes the outcome of that particular game and pays no attention to all the other features. Validation accuracy is abysmal. To prevent the network from fingerprinting the team compositions I tried using dropout on the team inputs only. Whereas in my experience keep_prob around 0.8-0.9 usually yields the best results, here I saw the best results with keep_prob 0.35. This kind of works but it doesn't feel great to basically withhold 65% of player information from the network.

What's the best way to prevent this sort of "cheating" where the network overfits to a small subset of features and looks up memorized results?


texts: Getting low accuracy on keras pretrained word embeddings example

texts: Original blog post claims that it is possible to get 95% accuracy on the validation set (20 Newsgroup dataset) after only 2 epochs using pretrained word embeddings (glove.6B.100d). All code is located here. I did not changed anything in this example but getting only 40% accuracy on the validation set after 2 epochs and 75% after 10 epochs. I can't get to 95% accuracy even after 20 epochs. Switching from tensorflow to theano backend do not make any significant change. I'm using Keras 2.0.2, tensorflow-gpu 1.0.1, theano 0.9.0, python 2.7.12. What I'm doing wrong?


texts: Does it make sense to do train test split when trainning GANS?

texts:
For normal supervised learning the dataset is split in train and test (let's keep it simple).

Generative Adversarial Networks are unsupervised learning but there is a supervised loss function in the discriminator.

Does it make sense to split the data into train and test when training GANs?

My first opinion will be no, but I am not 100% sure. Is there any reason why having a test set will help the Generator?


texts:
I installed cuda-11.0 and cuDNN-8.2 and Tensor-RT 7.1. And then installed tensorflow-gpu. But tf.is_gpu_available() is showing False.
Actually, I installed tensorflow-gpu first. I have pip3 20.1.1 and Nvidia GForce 930MX and Ubuntu 20.04.
Here is the nvidia-smi command output:


texts:
I have highly imbalanced dataset. I am using XGBoost and I got the following results without balancing the dataset out:

Precision: 0.87
Recall: 0.79
F1: 0.83

My parameters are:

xgb_model=xgb.XGBClassifier(
    objective = "binary:logistic",
    n_estimators=500,
    max_depth=8,
    verbosity=2,
    random_state=42)
xgb_model.fit(X_train,y_train, eval_metric='aucpr', eval_set=eval_set)

If I balance out my dataset via SMOTE, I get the following results:

Precision: 0.81
Recall: 0.80
F1: 0.81

If I use an imbalanced dataset, how do I improve recall (via precision trade-off)?


texts: Identifying whether training dataset is imbalanced in CART

texts:
I'm new to machine learning domain. I want to build a Classification tree for Binary classification of data.

I have a training data set of 269 records out of which 56 records belong to 'yes' class and 213 records to 'no' class.

Is this data imbalanced for building CART model? Do I need to undersample 'no' class records? Also from Gini index, Chi-Square and Information Gain, which algorithm is best for node splitting?

P.S.:- I can't increase the size of the dataset further.


texts: Classifier training long time due to the size data

texts:
I have a problem to train my classifier.

I have 10 different kinds of music genres, each genre with 100 songs, after making an Mfccs I have a numpy array of (1293, 20)

If all together with np.vstack I have an array of (1293000, 20) and another for the labels.

When I run model.fit (features, labels), it takes a lot of time.

I have also tried with:

from sklearn.manifold import TSNE
X_embedded = TSNE (n_components = 2).fit_transform(features)
X_embedded.shape

I've tried to reduce the songs from 1000 to 100 but it's still taking a long time.

Any idea how I can classify songs with arrays with so much data?

I put some code:

scaler = sklearn.preprocessing.StandardScaler()
y, sr = librosa.load('EXAMPLE1')
mfcc = librosa.feature.mfcc(y, sr=sr, n_mfcc=20).T
mfcc_scaled = scaler.fit_transform(mfcc)
mfcc_scaled.shape # (1293, 20)

y, sr = librosa.load('/Users/josetorronteras/AnacondaProjects/Neural-Networks/genres/pop/pop.00044.au')
mfcc2 = librosa.feature.mfcc(y, sr=sr, n_mfcc=20).T
mfcc_scaled2 = scaler.fit_transform(mfcc2)
mfcc_scaled2.shape # (1293, 20)

tmp_arr = []
tmp_arr.append(mfcc_scaled)
tmp_arr.append(mfcc_scaled2)
mafcc_list = np.vstack(tmp_arr)

mafcc_list.shape # (2586, 20)
a0 = np.zeros(len(mfcc_scaled))
a1 = np.ones(len(mfcc_scaled2))

labels = np.concatenate((a0, a1))
labels.shape # (2586,)

Thanks


texts:
K-Means initializes the centroids randomly, but there are other methods to initialize. In this paper, http://ilpubs.stanford.edu:8090/778/1/2006-13.pdf, they propose randomly choosing a data point initially then choose the other centroids based on the distance from the initial centroid.

My question is: how does this give you the right result? Say my data clusters naturally into three clusters, a noisy cluster around the (x, y) points (1, 1), (0, 0), and (-1, -1). Say I use the method from the paper and initially choose a data point (1.32, 0.98) and mark it as the center of cluster #1. According to the paper, I choose the next centroid based on distance, so the next point will be around (-1, -1). Say the data point chosen for cluster #2 is (-1.12, -0.89). These first two steps make sense, but now I continue on to cluster #3 and again I chose based on distance so I'll end up putting another cluster center very close to cluster #2's center. What am I missing here? Shouldn't the centers be chosen based on the sum of distances from the already initialized cluster centers?

EDIT:
Initially, I randomly choose a data point to mark as center of cluster #1. I choose the red point. Now I calculate the distances between red point and all other data points and choose the furthest point away as center of cluster #2. This is the green point. My question is: according to the paper, I repeat this and calculate distances from the red point to all remaining points and take the furthest away, but this puts me back near the green point, but I was trying to get to the center cluster.


texts: Py error could not broadcast input array from shape (244,244,3) into shape (224,224,3) arrays have the same shape (224, 224, 3)

texts:
this similar to other questions where there is an error related to "broadcasting" a np array with a shape of (244, 244, 3) and a np array with a shape of (244, 244, 4) but in this case both arrays have the same shape (244, 244, 3) i am new if have done this incorrectly let me now thanks code is hear
import tensorflow.keras
from PIL import Image, ImageOps
import numpy as np

SuccsesCriteria = 0.5999999
global data
global model

def start():
    global data
    global model
    np.set_printoptions(suppress=True)
    model = tensorflow.keras.models.load_model('Asstets/keras_model.h5')
    data = np.ndarray(shape=(1, 224, 224, 3), dtype=np.float32)

def imageToArray(image):
    #size = (224, 224)
    #image = ImageOps.fit(image, size, Image.ANTIALIAS)
    # turn the image into a numpy array
    image_array = np.asarray(image)
    normalized_image_array = (image_array.astype(np.float32) / 127.0) - 1
    return normalized_image_array

def predictionToBool(_prediction):
    for _p in _prediction:
        if _p[0] >= SuccsesCriteria:
            print('Cat')
            return True
        elif _p[2] >= SuccsesCriteria:
            print('No Cat')
            return True
        elif _p[1] >= SuccsesCriteria:
            print('???')
            return False

def predictFromImage(image):
    global data
    global model
    data[0] = imageToArray(image)
    # run the inference
    return model.predict(data)

full error
2021-05-12 09:11:49.477186: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
WARNING:tensorflow:No training configuration found in the save file, so the model was *not* compiled. Compile it manually.
Traceback (most recent call last):
  File "/Users/***************/Keras/main.py", line 8, in <module>
    print(AI.predictFromImage(Cam.getCamImage()))
  File "/Users/***************/Keras/AI.py", line 44, in predictFromImage
    data[0] = imageToArray(image)
ValueError: could not broadcast input array from shape (244,244,3) into shape (224,224,3)


texts: print a specific column with a condition using pandas

texts:
I have a data set which contains 5 columns, I want to print the content of a column called 'CONTENT' only when the column 'CLASS' equals one.
I know that using .query  allows me to select a condition, but it prints the whole data set. I tried to drop the unwanted columns, but I finished up with unaligned and not completed data: -

CONTENT CLASS
0 Huh, anyway check out this you[tube] channel: ... 1
1 Hey guys check out my new channel and our firs... 1
2 just for test I have to say murdev.com 1
3 me shaking my sexy ass on my channel enjoy ^^ \xef\xbb\xbf 1
4 watch?v=vtaRGgvGtWQ Check this out .\xef\xbb\xbf 1
5 Hey, check out my new website!! This site is a... 1
6 Subscribe to my channel \xef\xbb\xbf 1
8 You should check my channel for Funny VIDEOS!!\xef\xbb\xbf 1
9 and u should.d check my channel and tell me wh... 1
10 Hey subscribe to me\xef\xbb\xbf 1
11 Once you have started reading do not stop. If... 1
12 https://twitter.com/GBphotographyGB\xef\xbb\xbf 1
13 subscribe like comment\xef\xbb\xbf 1
14 please like :D https://premium.easypromosapp.c... 1
15 Hello! Do you like gaming, art videos, scienti... 1
17 http://www.ebay.com/itm/171183229277?ssPageNam... 1
18 http://ubuntuone.com/40beUutVu2ZKxK4uTgPZ8K\xef\xbb\xbf 1
19 We are an EDM apparel company dedicated to bri... 1
21 subscribe to my channel people :D\xef\xbb\xbf 1
22 Show your AUBURN PRIDE HERE: http://www.teespr... 1
24 CHECK OUT MY CHANNEL 1
25 marketglory . com/strategygame/andrijamatf ear... 1
26 Hey guys! Im a 12 yr old music producer. I mak... 1
27 Check me out! I\'m kyle. I rap so yeah \xef\xbb\xbf 1
29 Subscribe to me for free Android games, apps.. \xef\xbb\xbf 1
30 everyone please come check our newest song in ... 1
32 sub my channel for no reason --\xef\xbb\xbf 1
33 Check out my dubstep song "Fireball", made wit... 1
36 Check my channel please! And listen to the bes... 1
37 SUB 4 SUB PLEASE LIKE THIS COMMENT I WANT A SU... 1
.. ... ...
297 Please help me go to college guys! Thanks from... 1
298 https://www.facebook.com/SchoolGeniusNITS/phot... 1
299 I am so awesome and smart!!! Sucscribe to me!\xef\xbb\xbf 1
300 Follow 4 Follow @ Va... 1
301 http://hackfbaccountlive.com/?ref=4436607 psy... 1
302 https://www.facebook.com/nicushorbboy add mee ... 1
303 im sorry for the spam but My name is Jenny. I ... 1
305 please throw a sub on my channel\xef\xbb\xbf 1
307 Go to my channel if u want to see a fly gettin... 1
309 COME AND CHECK OUT MY NEW YOUTUBE CHHANEL, GOI... 1
311 Please check out my vidios guys\xef\xbb\xbf 1
313 Hey guys can you check my channel out plz. I d... 1
315 PLEASE SUBSCRIBE ME!!!!!!!!!!!!!!!!!!!!!!!!!!!... 1
317 Check out my Music Videos! Fuego - U LA LA Rem... 1
318 Check out my Music Videos! and PLEASE SUBSCRIB... 1
319 www.marketglory.com/strategygame/lordviperas\xef\xbb\xbf 1
323 Limit sun exposure while driving. Eliminate th... 1
325 http://hackfbaccountlive.com/?ref=5242575\xef\xbb\xbf 1
327 https://www.facebook.com/FUDAIRYQUEEN?pnref=st... 1
329 FOLLOW MY COMPANY ON TWITTER thanks. https:/... 1
331 Hey come check us out were new on youtube let ... 1
333 Look at the pictures, if not difficult http://... 1
335 Hey guys can you check my YouTube channel I kn... 1
337 https://www.facebook.com/tofikmiedzynB/photos/... 1
338 https://www.facebook.com/eeccon/posts/73394924... 1
339 http://www.bing.com/explore/rewards?PUBL=REFER... 1
340 Please do buy these new Christmas shirts! You ... 1
341 Free my apps get 1m crdits ! Just click on the... 1
347 subscribe to me for call of duty vids and give... 1
348 hi guys please my android photo editor downloa...


texts: Using reinforce algorithm with per-action reward instead of per-trajectory reward

texts:
I've found some articles that talk about the reinforce algorithm / monte carlo method. The algorithm boils down to using this equation.

The right summation over the trajectory is the reward for the trajectory. This is multiplied by the derivative of the log likelihood of the action sequence. The idea is you take a large sample of trajectories and average them with this equation to estimate the policy gradient.

I want to adapt this core algorithm to use discounted rewards for actions within the trajectory instead of using a total reward for trajectories. I read somewhere that this can reduce variance and is a common optimization. Could someone please derive how the action-reward would work or cite a source.


texts: How to add bias consideration into logistic regression code?

texts:
I am a complete beginner in coding and machine learning, and I've been tasked with learning what's under the hood of logistic regression (so I have pieced together the python code below) but I've been asked to figure out how to add bias into this code. I'm completely stuck on at what point it would need to be added into, I think at the point I am defining the hypothesis function - but if anyone would be able to point me in the right direction to figure this out I would really appreciate it.

If it helps, this logistic regression is being used to classify if a tumour is benign of malignant with the wisconsin breast cancer dataset (https://www.kaggle.com/uciml/breast-cancer-wisconsin-data)

X_train,X_test,Y_train,Y_test = train_test_split(X,Y,test_size=0.3)

X = data["diagnosis"].map(lambda x: float(x))

X = data[['texture_mean','perimeter_mean','smoothness_mean','compactness_mean','symmetry_mean', 'diagnosis']]
X = np.array(X)
X = min_max_scaler.fit_transform(X)
Y = data["diagnosis"].map(lambda x: float(x))
Y = np.array(Y)

def Sigmoid(z):
    if z < 0:
        return 1 - 1/(1 + math.exp(z))
    else:
        return 1/(1 + math.exp(-z))
def Hypothesis(theta, x):
    z = 0
    for i in range(len(theta)):
    z += x[i]*theta[i]
return Sigmoid(z)enter preformatted text here

def Cost_Function(X,Y,theta,m):
sumOfErrors = 0
for i in range(m):
    xi = X[i]
    hi = Hypothesis(theta,xi)
    error = Y[i] * math.log(hi if  hi >0 else 1)
    if Y[i] == 1:
        error = Y[i] * math.log(hi if  hi >0 else 1)
    elif Y[i] == 0:
        error = (1-Y[i]) * math.log(1-hi  if  1-hi >0 else 1)
    sumOfErrors += error

const = -1/m
J = const * sumOfErrors
print ('cost is: ', J )
return J

def Cost_Function_Derivative(X,Y,theta,j,m,alpha):
sumErrors = 0
for i in range(m):
    xi = X[i]
    xij = xi[j]
    hi = Hypothesis(theta,X[i])
    error = (hi - Y[i])*xij
    sumErrors += error
m = len(Y)
constant = float(alpha)/float(m)
J = constant * sumErrors
return J

def Gradient_Descent(X,Y,theta,m,alpha):
new_theta = []
constant = alpha/m
for j in range(len(theta)):
    CFDerivative = Cost_Function_Derivative(X,Y,theta,j,m,alpha)
    new_theta_value = theta[j] - CFDerivative
    new_theta.append(new_theta_value)
return new_theta

initial_theta = [0,1]
alpha = 0.01
iterations = 1000
Logistic_Regression(X,Y,alpha,initial_theta,iterations)


texts: How to interpret Hashingvectorizer representation?

texts:
I cannot really understand the logic behind Hashingvectorizer for text feature extraction. I can follow the logic of Bag of Word or TFiDF where the features are values for all/certain words/N-grams per document and as such one can compute (dis)similarity between the representation vector.

But how could one imagine the arbitrary number of features in case of Hashing?
Are these features purely synthetic features with no correlation of the original words?
Is it able to learn from some training set (by "fit") and apply this learned features to new sentences (by "transform") like other vectorizers?
Does Hashing represent any semantic distance like word2vec for words? How?


texts: What's the difference between Error, Risk and Loss?

texts: When we talk about 'Minimizing Loss', we often talk about loss functions such as Mean Squared Error (MSE); the term 'Empirical Risk Minimization' is often used interchangeably. So what's the difference between Error, Loss and Risk?


texts: which metric is better for boosting methods

texts:
I work on a dataset of 300 000 samples and I try to make a comparison between logistic regression (with gradients descent) and a LightBoost for binary classification in order to choose the better one.

I want to know in this case which metric should I use it and WHY?

Accuracy ??
AUC Test value ??
RMSE  ??
LogLoss ??


texts: Low accuracy using Multi Layer perceptron network on CIFAR10 dataset?

texts:
I  trained Muti layer perceptrons using keras to classify cifar-10 dataset the results I got shows that there is something wrong in the code ,because all the epoch are identical here is the code :
In this I flatten the data from 32x32x3 to 3072 array.

   #nn layers
import numpy
from keras.datasets import cifar10
from keras.models import Sequential
from keras.layers import Dense
from keras.layers import Dropout
from matplotlib import pyplot
from scipy.misc import toimage

#load data
(Xt,Yt),(Xts,Yts)=cifar10.load_data()

num_pix=Xt.shape[1]*Xt.shape[2]*Xt.shape[3]
X_tr=Xt.reshape(-1,3072).astype('float32')
X_ts=Xts.reshape(-1,3072).astype('float32')
#feature scaling
X_tr= X_tr / 255
X_ts = X_ts / 255
#one hot encoding
from keras.utils import np_utils
Ytr=Yt.reshape(-1,50000)
Yt=np_utils.to_categorical(Ytr)
Yts=Yts.reshape(-1,10000)
Yts=np_utils.to_categorical(Yts)
num_class=Yts.shape[1]

def base_model():
    model=Sequential()
    #build layers Dense(10, init="normal", activation="relu"
    model.add(Dense(num_pix,input_dim=num_pix,activation='relu'))#input layer
    model.add(Dense(num_class,activation='softmax'))#output layer
    #compile model
    model.compile(loss='squared_hinge',optimizer='adam',metrics=['accuracy'])
    return model
    #build the model
model=base_model()
#Fit model
model.fit(X_tr,Yt,validation_data=(X_ts,Yts),epochs=10,batch_size=200,verbose=2)
#Final assesment
score=model.evaluate(X_ts,Yts,verbose=0)
print("Baseline Error: %.2f%%" % (100-score[1]*100))

  Results :

 Train on 50000 samples, validate on 10000 samples
    Epoch 1/10
    134s - loss: 0.9900 - acc: 0.1000 - val_loss: 0.9900 - val_acc: 0.1000
    Epoch 2/10
    97s - loss: 0.9900 - acc: 0.1000 - val_loss: 0.9900 - val_acc: 0.1000
    Epoch 3/10
    99s - loss: 0.9900 - acc: 0.1000 - val_loss: 0.9900 - val_acc: 0.1000
    Epoch 4/10
    96s - loss: 0.9900 - acc: 0.1000 - val_loss: 0.9900 - val_acc: 0.1000
    Epoch 5/10
    96s - loss: 0.9900 - acc: 0.1000 - val_loss: 0.9900 - val_acc: 0.1000
    Epoch 6/10
    96s - loss: 0.9900 - acc: 0.1000 - val_loss: 0.9900 - val_acc: 0.1000
    Epoch 7/10
    96s - loss: 0.9900 - acc: 0.1000 - val_loss: 0.9900 - val_acc: 0.1000
    Epoch 8/10
    95s - loss: 0.9900 - acc: 0.1000 - val_loss: 0.9900 - val_acc: 0.1000
    Epoch 9/10
    95s - loss: 0.9900 - acc: 0.1000 - val_loss: 0.9900 - val_acc: 0.1000
    Epoch 10/10
    95s - loss: 0.9900 - acc: 0.1000 - val_loss: 0.9900 - val_acc: 0.1000
    Baseline Error: 90.00%

Any help will be appreciated.can my model be overfitted?


texts: Doing my first Gaussian Process Regression using scikit-learn: Something's wrong with my Covariance function

texts:
I'm trying to do my first Gaussian Process Regression using scikit-learn.
Here's my MWE:
from sklearn.gaussian_process import GaussianProcessRegressor
from sklearn.gaussian_process.kernels import Matern

kernel=Matern(nu=0.5)
model=lambda x: x*np.sin(x)
xdata=np.array([1,3,5,6,8])
ydata=model(xdata)
xfit=xdata[:,np.newaxis]
gp=GaussianProcessRegressor(kernel=kernel,alpha=0.5)
gp.fit(xfit,ydata)

When I run the above, I get
(...)\sklearn\gaussian_process\kernels.py:1623: RuntimeWarning: invalid value encountered in true_divide
  K_gradient = K[..., np.newaxis] * D \

Why is that?


texts: Problem while reusing model for new observations

texts:
I want to  build a model using R, save it and reloading it when required. I am able to save the model and reload it without any problem, however when i use the model along with predict statement, I am getting errors.

This is what i do:

  set.seed(345)
  df <- data.frame(x = rnorm(20))
  df <- transform(df, y = 5 + (2.3 * x) + rnorm(20))

  model

  m1 <- lm(y ~ x, data = df)

  save this model

  save(m1, file = "my_model1.rda")

a month later, new observations are available:

  newdf <- data.frame(x = rnorm(20))

  load the model

  load("my_model1.rda")

When i use the above model to predict for new values, I am getting output values as "NA"

  predict(m1, newdata = newdf)

"NA"  "NA"  "NA"  "NA"  "NA"  "NA" "NA"  "NA"  "NA"

I could able to see that the model has been loaded correctly by giving a print command

  print(m1)

Call:
lm(formula = y ~ x, data = df)

Coefficients:
(Intercept)            x
      4.792        2.119

Is there anything that I miss here?


texts: Is there a data structure that works similarly to a (binary) tree?

texts: Im currently working with some data, which is part of a very large data set (height, about 6000 elements per timestep - timestep every minute over days at a time). I need to differentiate between different "layers" in the data per timestep - which im doing with a binary tree, writing a programme in python. The layers are defined by a coefficient, which is part of the data set. The coefficient stays more or less constant when the material is constant, and so when the coefficient changes, it implies a layer ie change in material. So, I take the whole height range (node 0), and find the smallest coefficent in this range. This then splits the layer in 2 - nodes 1 and 2. I then take the range of node 1, and do the same thing (finding the minimum coefficient), which gives me nodes 3 and 4. And so on and so on. It works well, and I have my head wrapped around it quite well. I just got thinking today - is there another data structure that exists that could achieve a similar thing/works in a similar way?


texts: Error: keras merge LSTM layers in sum mode

texts:
I want to merge two sequential models in sum mode into one model using keras as:

left = Sequential()
left.add(LSTM(64,activation='sigmoid',stateful=True,batch_input_shape=(10,look_back,dim)))
right = Sequential()
right.add(LSTM(64,activation='sigmoid',stateful=True,batch_input_shape=(10,look_back,dim)))
model = Sequential()
model.add(Add()([left, right]))

But the statement model.add(Add()[left,right]) gives error: Layer add was called with an input that isn't a symbolic tensor. Received type: . Full input: [, ]. All inputs to the layer should be tensors.


texts: Why Adaboost SAMME needs f to be estimable?

texts:
I am trying to understand the mathematics behind SAMME AdaBoost:
At some stage, the paper adds a constraint for f to be estimable:

I do not understand why this is required. Can someone explain a bit better why this restriction is needed?
As well, would be possible to use a different constraint than the one added in the paper that would make f estimable?


texts: Extracting the time duration for events from an event log

texts: I have an event file (very similar to a log file) that logs event information that includes user data and time stamps. The events have an ordering for each user. So event E1 occurs before E2 that occurs before E3 and so on. For each user, I need to calculate the time durations to transition from E1 to E2 and so on. The event file has a line for each event. Naively, to do this I can group by the user and event time (for example in pandas) and then parse the group by result line by line. For each new line, I can calculate the time duration. Since it appears that this should be a fairly common data extraction task. Is there a standard way to do this rather than how I have described this above?


texts:
I'm working with an LSTM network to predict the surface roughness due to biogrowth on ships. I've got a Network that fits the input data I have relatively well, the problem is when I'm using it to predict future roughness values.
I've 3 case scenarios - one where the hull is not cleaned, second where the hull is cleaned and the third where the hull is repainted. Based on the inputs - the idea is the trend drops slightly when the hull is cleaned and considerably when it is repainted.
However when I use the same inputs to predict the roughness for the different cases, the outputs differ in the region where it is supposed to be the same, i.e it is predicting the trend to go upward in the case where the hull is cleaned, but when it is not cleaned it doesn't show this upward trend.
I assume it is seeing the pattern and seeing that as there is a future cleaning - the trend must go up. But this isn't reality. I've tried swapping the LSTM for a basic NN but this still exists.
Any help would be useful.


texts: Interactive labeling/annotating of time series data

texts:
I have a data set of time series data. I'm looking for an annotation (or labeling) tool to visualize it and to be able to interactively add labels on it, in order to get annotated data that I can use for supervised ML.

E.g. the input data is a csv-file and the output is another csv-file of the format timestamp,label.

Therefore I need something like this:

to visualize data
to select a specific area
output the labels with timestamps

As an example:

Building such a tool in python will not take too long, however I was just wondering how other people solve this problem and maybe there are already nice OS tools for doing this. Thank you!


texts: tensorboard showing the epoch loss and accuracy for validation data but not training data

texts:
I am doing binary classification using CNN and viewing the results using tensorboard. The problem is that the tensorboard is not showing the loss and accuracy for training data.

As seen curves for training data are not shown


texts: Best Loss function to use for Multiple Categories which have an implicit order

texts:
I am wondering what options I have for loss functions when the task at hand is Multi-Class Classification, where the classes themselves have an implicit order, ranging from least popular (class 0) to most popular (class 11). The options I have thought of are the following:

Using regular Categorical Crossentropy Loss
Making the model a regression model based on the class number itself

Is there a categorical loss function which takes this implicit order between classes into account, or should I just try the two options listed above?


texts: what are the next step after ML prediction and how to proceed?

texts:
I have trained an ML model with a good accuracy but what next?

I am facing difficulty in answering this question, how will you present your model.
Which framework do  you use
How do you make sure the model is learning continuously.
Where can I find definite guide for this approach?

I am good at building and predicting ML, DL & NLP. But stuck to deploy further.


texts: Features selection in imbalanced dataset

texts:
I have some doubts regarding an analysis. I have a dataset with class imbalance. I am trying to investigate some information from that data, e.g., how many urls contain http or https protocols.
My results are as follows:
http in dataset with class 1: 10
http in dataset with class 0: 109
https in dataset with class 1: 180
https in dataset with class 0: 1560

I am trying to build a classifier based on some features and the presence of protocols was supposed to be taken into account.
However, on the basis of the above results, what do you think I should say?
Does it make sense to say that the most websites having class 0 have an https protocol, even if I have a dataset with class imbalance?
For a model, I would consider resampling techniques. Should I work on this analysis (so make this conclusion) after the resampling, or it would make sense to check features importance with other tests (e.g., Pearson correlation, if it is appropriate in this case)?
Any suggestion would be greatly appreciated it.


texts: Tips and tricks for designing time-series variational autoencoders

texts:
I am new to VAEs but find them quite fascinating. I was wondering if anyone might have any tips or tricks regarding, how one should build the encoder and decoder layers w.r.t. to time-series data.

Suppose my inputs are $x \in \mathbb{R}^{1\times1000}$ so they have a thousand dimensions. The data is highly-nonlinear and rather noisy. I have a couple of tens of thousand $x$:s.

With that in mind, what are some general guidelines (because I cannot find any, owing to the VAE field being somewhat particular and very new):

How deep should my encoder and decoder network be? Are there any good guidelines?
Should one use fully connected dense networks, or stacked conv1Dnets?
What activation functions are good choices?
Can we say anything about the 'best' dimensionality of the latent dimension?
Like everyone else I imagine, my loss function is the sum of a reconstruction term, and the KL divergence regularization term. Is there something else one should consider?
Batch normalization? Currently, on my problem, it doesn't make a blind bit of difference. But it should. Should one always use batch-norm?
In the decoder layer, is it better to up-sample before returning to the original input dimension?

Any tips or tricks would be most welcome. I have naturally tried a few of these, but I cannot discern a trend at present.


texts:
I built a MLP  for  classification problem. I used KDD99 dataset. but the problem is that when I run it it gives ech time a different value of accuracy.
Is this logic? I mean could the same model for the same data gives each time a different value of accuracy? if so why?


texts: Is it correct to join training and validation set before inferring on test-set?

texts: I would like to know if is a correct procedure to join training-set and validation-set together, in order to train the model on this new dataset, before making predictions on the test-set.


texts:
I struggle to understand the mathematical motivation for the binary classification model calibration curve.
Why do we assume that the predicted probabilities should be consistent with the proportion of 1's in the probability bin (# of 1's in the bin)/(total # of samples in the bin) ? It's obvious for Decision Tree where the (# of 1's in the bin)/(total # of samples in the bin) ratio is explicitly the model output, but how is this related to the other models? How to show that this ratio and predicted probabilities should be equal?