texts: How are clusters from DBSCAN sometimes non-convex?

texts:
I've been using clustering in my bag of ML techniques for quite some time now, and  I've never found a satisfying answer to this question.

In DBSCAN, we define a maximum radius with which to form clusters.  The algorithm will scan the space and group together points that are ALL reachable from one another.  However, we can sometimes end up with a non-convex cluster.

My confusion is around how the notion of a "radius", which describes a convex object, can be an input to an algorithm which results in a non-convex object?


texts: term for data that compares a day to same day of a week/month/year ago

texts: What is the technical term for a report or dataset that compares data of an interval of time with that of the same interval in a previous week, month, or year?


texts: How many features should I take out of 515 features?

texts:
In continuation to this question, I have a conceptual question. If I am not using the 'nfeatures' then I am getting 7 features from a feature set of 515 features. If I use the 'nnfeatures' tag (this tag specifies the minimum number of features I want it to display) with a value of 10, then I am getting those 7 faetures along with 3 new one. Now I am unable to decide how many features should I take. Is it okay to take 7 features out of 515 features? Will any ML algorithm be able to capture the characteristics of the data with just these 7 features?

Another question is, a criterion value is given when the sequentialfs algorithm is applied in MATLAB. What is this criterion function? From this page I came to know that a set of features are chosen whose mean criterion value is the lowest. So does it mean that only those features are chosen with the minimum criterion value? Also, if this is the case, then how does the algorithm decide how many features it should select?


texts: Need help on pandas data frames filtering on a csv file

texts:
Raw data gets generated daily
We need to get
success rate for current month on a daily basis &
success rate of each cell by end of that month from Jan till current month.

Raw data has following columns:

Column Names    example values
BackupClient    an4lsbk0304.an.ppp.com
BackupDriver    NetBackup
BackupMaster    an4lsbk0300.an.ppp.com
BackupPolicyID  an.72699
BackupPolicyName    ancfsv02a_aggr1_ancfs02n02b_L_an_vlw_atom_001
Cell    an
LastFullResult  Failure
LastFullStartTime   4-10-19
PolicyType  NFS
Status  Active
LastFullExitCode    96

Each column – “Cell”
I need to find following result columns:

Oct Full ran, failed & not completed yet
Oct Full not ran
Oct full ran successful
Grand Total
Success rate for full ran in Oct
Success rate of full backup
Percentage of backup coverage

Oct Full ran, failed & not completed yet:
Oct full ran, failed & not completed yet means –
“LastFullStartTime” – contains current month date and non-empty     &&
“LastFullResult” – Failed                       &&
“status code” – greater than 1
Oct Full not ran yet:

Oct Full not ran means –
“LastFullStartTime” - empty or date older than current month

Oct full ran successful –
Oct full ran successful means –
“LastFullStartTime” – contains current month date           &&
“LastFullResult” – success
Grand Total
Grand Total means –
Count of BackupPolicyID for each distinct cell; should be ideally equal to above 3 columns (1+2+3=4)
Success rate for full ran in Oct
Success rate for full ran in Oct means –
Above column1/(column1+column3) in percentage
Success rate of full backup
Success rate of full backup means –
Above column1/(column1+column2+ column3) in percentage
Percentage of backup coverage
Percentage of backup coverage means –
Above (column1+column3)/column4

code:

import pandas as pd
import os
RD = pd.read_csv("C:/Users/acharbha/Desktop/fullbackup_success/python/raw_Data_success_Rate.csv")
print(RD.info())


texts: Splitting binary classification into smaller susbsets

texts:
As an example. If you are tying to classify humans from dogs. Is it possible to approach this problem by classifying different kinds of animals (birds, fish, reptiles, mammals, ...) or even smaller subsets (dogs, cats, whales, lions, ...)

Then when you try to classify a new data set, anything that did not fall into one of those classes can be considered a human.

If this is possible, are there any benefits into breaking a binary class problem into several classes (or perhaps labels)?

Benefits I am looking into are: accuracy/precision of the classifier, parallel learning.


texts: Choosing between prepadding and postpadding for variable length sequence data in a CNN

texts:
Convolutional Neural Networks (CNN) assume the same sized data. Often sequence data is variable length (e.g., natural language). Thus sequence data is padded to make sure all data is all the same length.

Prepadding is adding zeros to the beginning of shorter sequences. Postpadding is adding zeros to the ending of shorter sequences.

What is the empirical reason to choose prepadding or postpadding?


texts: Please help to identify if there is meaning of linear variables separation by features in phase space?

texts:
Description:

I have clustering problem with next input:

I have truth dataset containing labels for each row of features.
I've found that some features can be modified to some phase space,
where they are clearly distinguishable
(http://i.prntscr.com/iceOaKW8SdiIs-QWT8UVqg.png) only in case if I select subset containing only a few clusters (n_clusters << n_labels) and no other clusters together within this truth dataset. And for the close items (within space of modified features) it would be harder to distinguish them, lines become closer and closer.

Questions:

Does this separability give an actual knowledge, allowing to build the model or it's the false way of data understanding (does it make sense to consider using this parameter for the model teaching)?
If this separability makes sense and can be interpreted, which model will work better for this situation, does it makes sense to teach the model on the small subsets with distant items and join them somehow?

UPD:
According to my experiments, DBSCAN had shown that this parameter only created additional confusing, as on more rows in the data lines became denser and deviance within one line became larger than the distance between lines.


texts: Why RMSProp converges faster than Momentum?

texts:
Why is RMSProp in many cases converging faster than Momentum?

Momentum:

$$v_{dW} := \beta v_{dw} +(1-\beta)dW$$
$$W := W-\alpha v_{dw}$$

RMSProp:

$$ S_{dw} := B \cdot S_{dw} + (1-B)\cdot (dW)^2$$
  $$W := W- \alpha \frac{dW}{\sqrt{S_{dw}}}$$

Where $\alpha$ is the learning rate (0.01 etc), $\beta$ is the momentum term (0.9 etc), similar to B

From my point of view, both momentum and RMSProp have "tendency to keep moving". Well, I can see how RMSprop will naturally accelerate on flat surfaces due to

$$\frac{1}{\sqrt{S_{dw}}}$$

when $S_{dw}$ is small, but is there another benefit that RMSprop provides?


texts: Script to convert non-numeric elements that are numbers to numeric

texts:
I'm self-teaching myself some scikit-learn and tensorflow right now, and trying to get better at cleaning data since the rest seems reasonably straightforward.

I downloaded some Google Play Store data from here, and upon taking a peak, it appears to be primarily numerical, although the numbers stored are not of a recognized type.

dat.info()
returns

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 10841 entries, 0 to 10840
Data columns (total 13 columns):
App               10841 non-null object
Category          10841 non-null object
Rating            9367 non-null float64
Reviews           10841 non-null object
Size              10841 non-null object
Installs          10841 non-null object
Type              10840 non-null object
Price             10841 non-null object
Content Rating    10840 non-null object
Genres            10841 non-null object
Last Updated      10841 non-null object
Current Ver       10833 non-null object
Android Ver       10838 non-null object
dtypes: float64(1), object(12)
memory usage: 1.1+ MB

A lot of these are obviously numerical anyway when checking with head(), but some entries have something like "3.0M" in place of 3000000, and so I can't have any fun with them yet (I'm trying to first make histograms of them to explore). I wrote this quick loop to try and address this by converting them to a string, taking off the "M", then making them floats, but it doesn't seem to be working because the output still isn't numeric.

k=0
for i in dat["Reviews"]:
    #make the entry a string:
    j=str(i)
    #remove the Ms- I'm assuming these mean million
    if j[-1]=='M':
        j=j[:-1] #cut off last letter
        j=float(j)
        j=j*10**6
    dat[k,"Reviews"]=j
    k=k+1

There is likely a far easier way to do this that I'm missing, but the fact not even this works is strange. Does anyone have any ideas on how to deal with an unknown object type in a dataframe like this?


texts: Where exactly does $\geq 1$ come from in SVMs optimization problem constraint?

texts:
I've understood that SVMs are binary, linear classifiers (without the kernel trick). They have training data $(x_i, y_i)$ where $x_i$ is a vector and $y_i \in \{-1, 1\}$ is the class. As they are binary, linear classifiers the task is to find a hyperplane which separates the data points with the label $-1$ from the data points with the label $+1$.

Assume for now, that the data points are linearly separable and we don't need slack variables.

Now I've read that the training problem is now the following optimization problem:

${\min_{w, b} \frac{1}{2} \|w\|^2}$
s.t. $y_i ( \langle w, x_i \rangle + b) \geq 1$

I think I got that minizmizing $\|w\|^2$ means maximizing the margin (however, I don't understand why it is the square here. Would anything change if one would try to minimize $\|w\|$?).

I also understood that $y_i ( \langle w, x_i \rangle + b) \geq 0$ means that the model has to be correct on the training data. However, there is a $1$ and not a $0$. Why?


texts:
Does anyone know how to get the learning rate from participant data?

I'm computing all the expected values for all trials (=200)

$$V_t(S) = V_{t-1}(S)+ \alpha \cdot \text{error }_t $$

$$(\text{error }_t = R_t − V_{t-1}(S))$$

and then computing alpha at each trial using

$$\alpha= (V_t(S)  - V_{t-1}(S) )/ \text{error }_t$$

but it seems not the right way!

Suggestions?


texts: Transcription of Hindi text to English text

texts:
I am working on Hindi speech.

I want to convert Hindi text to English language text in Python. I am not interested in translation.

Example of the desired effect: input 'मेरा नाम मोहन है' to output 'mera naam mohan hai'.


texts: How to refit GridSearchCV on Multiclass problem

texts:
I'm trying to use GridSearchCV for my Multiclass problem. For starters, wanted to test it on KNeighborsClassifier.
First, here's the code where I define the function which uses GridSearchCV:
from sklearn.model_selection import GridSearchCV
from sklearn.model_selection import KFold

def grid_search(estimator, parameters, X, y):
    scoring = ['accuracy', 'precision', 'recall']
    kf = KFold(5)

    clf = GridSearchCV(estimator, parameters, cv=kf, scoring=scoring, refit="accuracy", n_jobs=-1)
    clf.fit(X, y)

    i = clf.best_index_
    best_precision = clf.cv_results_['mean_test_precision'][i]
    best_recall = clf.cv_results_['mean_test_recall'][i]

    print('Best score (accuracy): {}'.format(clf.best_score_))
    print('Mean precision: {}'.format(best_precision))
    print('Mean recall: {}'.format(best_recall))
    print('Best parametes: {}'.format(clf.best_params_))

    return clf.best_estimator_

And, here's where I use it, when I try running a K nearest neighbors classifier:
from sklearn.neighbors import KNeighborsClassifier

parameters = {'n_neighbors': [1, 2, 5, 10], 'weights': ['uniform', 'distance'], 'metric': ['manhattan', 'euclidean', 'chebyshev']}

knn = grid_search(KNeighborsClassifier(n_jobs=-1), parameters, X_train, y_train)

Under the current state of the above code I'm getting the following ValueError
ValueError: Target is multiclass but average='binary'. Please choose another average setting, one of [None, 'micro', 'macro', 'weighted'].

As you may have guessed, this might be related to the value of the refit parameter for GridSearchCV which currently is set to refit="accuracy" and this cannot work because the problem is multiclass.
I changed it's value many times, tried True or other explicitly stated metrics and nothing fixed the problem. On some of those tries, the error message changed to:

ValueError: For multi-metric scoring, the parameter refit must be set
to a scorer key or a callable to refit an estimator with the best
parameter setting on the whole data and make the best_* attributes
available for that metric. If this is not needed, refit should be set
to False explicitly.

Any advice?


texts: reliability of human-level evaluation of the interpretability quality of a model

texts:
Christoph Molnar, in his book Interpretable Machine Learning, writes that

Human level evaluation (simple task) is a simplified application level
evaluation. The difference is that these experiments are not carried
out with the domain experts, but with laypersons. This makes
experiments cheaper (especially if the domain experts are
radiologists) and it is easier to find more testers. An example would
be to show a user different explanations and the user would choose the
best one.

(Chapter = Interpretability , section = Approaches for Evaluating the Interpretability Quality).
Why would anyone pick/trust a human-backed(not an expert) model over, say a domain-expert backed model or even a functionally evaluated model(i.e. accuracy/precision/recall/f1-score etc. are considerably good)?


texts:
I am predicting number of vehicles in 4 traffic junctions.

So, I have following columns in my dataset :

DateTime
Junction_ID
Number_of_vehicles

At the first glance , this problem may look like Time series regression. But, the data given seems like Linear Regression problem.

So, I have applied linear regression in the following manner :

Used get_dummies extensively for all the columns. I used dummy variables for 31 days,24 hours ,7 days of weeks and 4 Junction Ids.
Then applied Linear Regression model

from sklearn.model_selection import train_test_split

x_train, x_test, y_train, y_test = train_test_split(train_data,train_vehicles)

clf.fit(x_train,y_train)

import math

pred=clf.predict(x_test)

pred.shape #got result as (12030,)

result = []
for x in pred:
result.append(math.ceil(x))

from sklearn.metrics import mean_squared_error

score=mean_squared_error(y_test, result)
rmse=math.sqrt(score)
print('RMSE is :', rmse)

I am getting RMSE value as 10.636853077462394

My questions are :

Since RMSE value is on lower side , can I say this model is decent ?
Is there any other approach which I can use on this dataset ?
Do I need to check for colinearity ?
How can I check if multiple variables are interrelated ?
Should I go for non-linear regression on this dataset ?


texts: how to avoid tokenizing w/ sklearn feature extraction

texts:
I'm trying to analyze some machine log files and the column I'm looking at can have values like 'Part.C1.11.Reading Status'. I want to treat the complete string as one token and I don't want it to be split into 'Part', 'C1', '11' and 'Reading# and 'Status'.

I've got the vague feeling that the token_pattern is the parameter I need to adjust so I tried to specify the beginning and the end of a string like so:

from sklearn.feature_extraction.text import CountVectorizer
cvo = CountVectorizer(token_pattern='^$',lowercase=False)
OriginCV = cvo.fit_transform(log['Message_Origin']).toarray()

However, the last line throws an error:
ValueError: empty vocabulary; perhaps the documents only contain stop words

I've also tried to explictly include dot and space in the token_pattern like so:

cvo = CountVectorizer(lowercase=False, token_pattern=r"(?u)\b\w\w+\b|\.|\s")

Throws no errors but does not do the trick (no change except for an additional token '.')

Not changing the default token_pattern does split the string at the spaces and colons though. I found this solution, which however modifies the string by removing e.g. the colons.
Any other idea how to solve this?

Thanks,
Mark


texts: How to Create Shared Weights Layer in Keras

texts:
I'm trying to set up a CNN in Keras, and I don't have much experience outside of simple NNs as far as the actual setup. I understand how the CNN is going to work on my data, but I need advice on how to set it up in Keras.

In my dataset, each datapoint is structured as follows. There are n columns, each of which is sensor data of people performing an action. There are 10 rows where each row is a single person's sensor data for that event. The label column is a score for how well they performed. Since each of their performances is related, each datapoint is a 10 by n matrix, and the label column for each data point is a 10 by 1 vector of scores.

Essentially, I want to set up my NN so that it treats:

Num|Sens0|Sens1|Sens2|Win
---|-----|-----|-----|---
A  |data |data |data | 0
B  |data |data |data | 0
C  |data |data |data | 0
D  |data |data |data | 0
E  |data |data |data | 0
F  |data |data |data | 0
G  |data |data |data | 0
H  |data |data |data | 1
I  |data |data |data | 0
J  |data |data |data | 0

the same as:

Num|Sens0|Sens1|Sens2|Win
---|-----|-----|-----|---
J  |data |data |data | 0
F  |data |data |data | 0
D  |data |data |data | 0
B  |data |data |data | 0
G  |data |data |data | 0
H  |data |data |data | 1
C  |data |data |data | 0
A  |data |data |data | 0
I  |data |data |data | 0
E  |data |data |data | 0

In my first layer (at the least), I want to set up convolutional neurons (I'm starting with 5, and moving on from there) so that each row shares weights amongst the observations, in such a way that the order I put the row in the datapoint doesn't matter (aside from it's correspondence with the label). Anyone know if this is possible? Otherwise I'll have to feed in each datapoint in all 10! orders, and I would prefer not to...

Any help is appreciated, thank you!


texts: What is the correct name for a stacked bar chart with a single bar (series), and many stacked segments?

texts:
I had a treemap that was better expressed in a linear format.
This resulted in a chart that I don't know the name of. It's something like a stacked bar chart, with just a single column/row:

(This example is a simplified version with less groupings)
It's similar to a bar chart, a treemap, a marimekko chart, etc., and some names come to mind based on these other charts, like:

Linear treemap
Stacked single bar chart
Linear segment chart

It's not a complicated chart, so I thought it would already have a succinct name. But I'm having trouble finding any kind of shorthand terminology for this chart format.


texts:
I'm a beginner in Spark and I want to calculate the average of number per name.

I have a JSON file with this information

df = spark.read.json("myjson.json")
df.select(avg(df["number"]), df["name"]).groupBy("name").show()

But I'm doing it wrong..

How can I solve my problem?

Thanks a lot


texts: How to extract a column that has the highest value within row in Hive?

texts:
I have a table, more or less in the following format

col1 col2 col3 ... col100
val1 val2 val3 ... val100

Where val* are doubles. Is there a way to extract for each row in which column is the highest value within row in Hive?

For example, for table like

col1 col2 col3
2     4   5
8     1   2

I would get

col3
col1


texts:
I am looking for code to implement a custom loss function instead of just classification error, or cross entropy for gradient boosted classification trees.
We are trying to model regime detection in financial markets based on economic indicators. The intended loss function is that if the model outputs a probability of each class of risk as p_low, p_medium, and p_high, then the loss function we are looking for is ( p_low * Returns(portfolio_low_risk) + p_medium * Returns(portfolio_medium_risk) + p_high * Returns(portfolio_high_risk) )
We already have a data frame of Returns(portfolio_i_risk).


texts:
Is it possible to perform Book index searching using Machine learning algorithms?
Inputs :
1 Book pages with page numbers as images.
2 Index words in the book.
Output: Tracing the page number/s with the indexes provided.


texts: I have some 14 years of work experience in operations management and education areas as Instructor of statistics and operations. I have little better command over statistical learnings. I do have a keen interest in data science learnings. Basically, want to perform a role of analyst on big data.  Will it right to make venture in Data Science field with this experience? There is touch on coding is with but do not project experiences.


texts: Recommended papers for deep learning based classification?

texts:
I'm working on a project where I have to build a classifier using Deep Neural Networks and I will be dealing with big data. I am familiar with neural networks, yet not an expert. I can obviously find a huge amount of literature online regarding deep learning but mostly will be convnets related.
Are there any good research papers for DNN-based classifiers or should I just stack more layers? I want to get some insight about the architecture and training process.
P.S: Using Deep Neural Networks is mandatory even if the accuracy is low, but it would be great if it worked out.


texts:
I have a lot of sentences (500k) which looks like this:

"Penalty missed! Bad penalty by Felipe Brisola  - Riga FC -  shot with right foot is very close to the goal. Felipe Brisola should be disappointed."
"Penalty saved! Damir Kojasevic  - Sutjeska Niksic -  fails to capitalise on this great opportunity,  shot with right foot saved  in the centre of the goal."
"Penalty saved! Stefan Panic  - Riga FC -  fails to capitalise on this great opportunity,  shot with right foot saved  in the centre of the goal."
"Penalty saved! Georgie Kelly  - Dundalk -  fails to capitalise on this great opportunity,  shot with right foot saved  in the centre of the goal."
"Penalty missed! Still  FC København 1, Crvena Zvezda 1. Marko Marin  - Crvena Zvezda -  hits the bar with a shot with right foot."

As you see, they are not really robotic, and after ending up writing 1500 lines of php code (with regex) and still being inconsistent, I decided to see my alternatives with machine learning.

What I am trying to achieve is:

For example this one:

"Penalty saved! Stefan Panic  - Riga FC -  fails to capitalise on this great opportunity,  shot with right foot saved  in the centre of the goal."

type => penalty
action => saved
reason => shot with right foot saved  in the centre of the goal
person => Stefan Panic

I stumbled upon spaCy and saw "Named Entity Recognition" and thought maybe I can use it for this purpose. Especially as I have huge training data.

I wanted to ask: Is spaCy's Named Entity Recognition is right for this task? If not, what should I try to learn for this task?

P.S: I know a little about python but nothing about ML


texts: How come powerbi does not SUM() properly?

texts:
I want to learn PowerBI so I downloaded it connected PowerBI to a MySQL database that uses this test data set:

http://learn.evermight.net/johnlai/demostore.sql

That data set has the tables invoice,invoice_item,customer,address,product.

I want to use PowerBI to give me a table of data that yields the same result as this query:

SELECT customer_id, CONCAT(first_name,last_name),
    (SELECT SUM(qty*unit_cost)
    FROM invoice_item WHERE invoice_id IN
        (SELECT invoice_id FROM invoice WHERE invoice.customer_id = customer.customer_id))
    AS PurchAmt
FROM customer
GROUP BY customer_id
ORDER BY PurchAmt DESC;

+-------------+------------------------------+-------------+
| customer_id | CONCAT(first_name,last_name) | PurchAmt    |
+-------------+------------------------------+-------------+
|          10 | YuriPreb                     | 13082458.70 |
|           7 | SylvaiBelange                | 13031101.01 |
|           9 | YmnuaKrish                   | 13004262.97 |
|           8 | AmnaChowdury                 | 12843937.12 |
|           3 | FlorenceIel                  | 12786294.27 |
|          11 | GordonBiyas                  | 12722387.44 |
|           5 | SabrinaFabo                  | 12675754.67 |
|           4 | AlishaMml                    | 12662484.31 |
|           1 | JayBobbin                    | 12537697.14 |
|          12 | SimulaUmmea                  | 12376986.77 |
|           2 | JayisonGam                   | 12313863.63 |
|           6 | FabriceLabe                  | 12176486.31 |
+-------------+------------------------------+-------------+

After connecting to the database with PowerBI, I defined two new columns in PowerBI:

Cost = 'demostore invoice_item'[qty]*'demostore invoice_item'[unit_cost]
FullName = CONCATENATE('demostore customer'[first_name],'demostore customer'[last_name])

Then I clicked the checkbox for fields customer.customer_id,customer.FullName,invoice_item.Cost and this is what I got:

Why are the results in PowerBI different from my SQL query?  For example, MySQL says Fabrice has 12176486.31 but PowerBI says 11286529.84 ?

UPDATE

I noticed that if I checkmark instead the following fields invoice.customer_id,invoice_item.Cost, and I do not summarize the invoice.customer_id as shown in this image here:

Then Customer 6 (which is Fabrice) shows the same amount as my MySQL query.  What's going on?


texts:
I’m trying to understand, and eventually build a Restricted Boltzmann Machine. I understand that the update rule - that is the algorithm used to change the weights - is something called “contrastive divergence”. I looked this up on Wikipedia and found these steps:

Take a training sample v, compute the probabilities of the hidden units and sample a hidden activation vector h from this probability distribution.
Compute the outer product of v and h and call this the positive gradient.
From h, sample a reconstruction v' of the visible units, then resample the hidden activations h' from this. (Gibbs sampling step)
Compute the outer product of v' and h' and call this the negative gradient.
...

I don’t understand step 3 and I’m struggling to grasp the concept of Gibbs sampling. Would someone explain this simply to me? I have covered neural networks if that helps you.


texts: finding maximum depth of random forest given the number of features

texts:
How do we find maximum depth of Random Forest if we know the number of features ?

This is needed for regularizing random forest classifier.


texts: How to import image data into python for keras?

texts:
I'm new to CNNs, starting off with keras. I'm currently using ImageDataGenerator to import my train/validation folders (which each have 2 class subfolders for my binary classification task). Was wondering how can I import my train/validation files without using ImageDataGenerator? I'm aware that ImageDataGenerator is good for accuracy as it does some augmentation, but I want to compare the accuracy to a training set without any augmentations. Essentially I think I need to put all the images into an array, but not sure how to.

Basically I want to know what is the normal way to import training/validation data for images, so I can compare what is the accuracy difference with/without imagedatagen. I know with normal NN tasks it's easy as you can just do pd.read_csv().

I'm currently importing like so:

train_datagen = ImageDataGenerator(
        rescale=1./255,
        shear_range=0.2,
        zoom_range=0.2,
        horizontal_flip=True)

test_datagen = ImageDataGenerator(rescale=1./255)

train_generator = train_datagen.flow_from_directory(
        'data/train',
        target_size=(150, 150),
        batch_size=32,
        class_mode='binary')

validation_generator = test_datagen.flow_from_directory(
        'data/validation',
        target_size=(150, 150),
        batch_size=32,
        class_mode='binary')


texts: Analysis of table to make compressing model

texts:
I have a table with hundred of entries in this style:
//HEADER//CODE1//decimals_here//CODE4//decimals_here//CODE3//decimals_here//CODE5//decimals_here//ENDING//
I want to make a simple text compressor that works for only 1 entry at each time. Can someone give me a kickstart hint to start playing in Orange?

Where do I split the text for better analysis? Can I find that also in Orange?
How can I get the most repeated/common segments and their occurrence?

Thanks


texts:
Recently I started reading more about NLP and following tutorials in Python in order to learn more about the subject. I started experimenting with words embeddings also, and I found some interesting results which I don't know how to interpret. I first used an English corpus for both training and testing and afterwards, I used the English corpus for training and a small French corpus for testing (all corpora have been annotated for the same binary classification task). In both cases, I used the pre-trained on tweets Glove embeddings. As the results in the case where I also used the French corpus improved (by almost 5%, reaching ~accuracy = 0.8), I was wondering if Glove was trained on multilingual data, as I didn't see anyone making this statement (I'm aware of the amount of data that is used) comparing to FastText, for example, where you have embeddings for different languages.

Also, if Glove supports multilingual information, this would also eliminate the need for mapping different embeddings into the same embedding space (as is the case for FastText).

Any clarification would be greatly appreciated.


texts: Keras Fit Function (R): Train Regression Model with multiple Labels

texts:
I am trying to implement a deep learning model in R using Keras. Let's say I had a dataset of people's faces and a CSV with information about the person's age, gender, and ethnicity. I want to train the model to predict a person's age from their photo.

For this, I have to use convolutional neural networks. Here is my pseudocode:

library(keras)

model <- keras_model_sequential()

model %>% ## define CNN model’s architecture.
## I will figure this out later

# configure model
model %>% compile(
    loss='mean_squared_error',
    optimizer='adam',
    metrics='mae'
)

history <- model %>% fit(
# train_array is a 300x100x100x1 array
# train_array stores the pixel values of 300 grayscale images
# of resolution 100x100
    x = train_array, y = ???,
    epochs = 10, batch_size = 30,
 )

My question is regarding the ??? marked above. I have three pieces of useful information to train the model: age, gender, and ethnicity. How do I add this information to the model? Again, I want the model to predict a single prediction of age.

I looked at the Keras fit function documentation but couldn't figure it out. It says that y is a "Vector, matrix, or array of target (label) data (or list if the model has multiple outputs)..."

EDIT:
That makes sense. Now the question is how do I set up the x list? Here's what I have so far:

trainFeatures <- list(pixels = train_array, gender = as.factor(trainGenders), ethnicity = as.factor(trainEthnicity))

history <- model %>% fit(
    x = trainFeatures, y = trainAge,
    epochs = 10, batch_size = 30,
)

I downloaded a toy dataset and set up a quick CNN model. I got the following error message:

Error in py_get_attr_impl(x, name, silent) :
  AttributeError: 'list' object has no attribute 'dtype'


texts: How do you think about neural networks and ways to design new models?

texts:
I'm currently learning about neural networks, and it seems to me that there usually is no good theoretical explanation given for why certain architectures work; there is most of the times, no formal argument as to why certain architectures work. The explanation give by professors seems to be more that we devise architectures based on what we think might work; not on some formalized notion. As an example, we think GRUs work because they allow us to remember information from the past, and allows us to forget and remember as we chose. Well, who's to say simple skip level connections don't allow that?
basically, is it the case that ANNs will remain blackboxes, and new techniques are basically architectures coming from gut feeling as to what might work?


texts: How can I combine images for Matlab deep learning?

texts:
I have a data set which consist a number of instances (of two-three different classes), and for each instance I have a handful of monochrome images. This handful of images have different resolutions (e.g. one 50x200 image, one 600x10 image, and one 128x128 image), but every instance has the same number of images and the exact same resolutions (so, e.g., for every instance there is one 50x200 image, one 600x10 image and one 128x128 image).

Now I would like to combine these images in some way, and use them to train a deep neural network. I know how to do this (in Matlab, using the Neural Network toolbox) on one image per instance at a time (e.g. only using the 50x200 images, ignoring the other images), but I have no idea how to proceed if I want to combine the images (which I hope will give the learning process more information, thus result in better classification performance).

Of course I could simply concatenate them in some way, e.g. I could stack them horizontally and fill empty areas with black pixels (thus creating one (50+600+128)x(200+10+128) = 778x338 image per instance). But not only would this create very large images (in my example, 8 times as many pixels as the sum of the number of pixels in the original three images), but more importantly it seems to me it would make little sense to apply the standard deep network constructs of convolution and pooling over such an images. First of all, the features that need to be discovered are likely to be very different in the different original images (and thus different parts of the new, large image), and second, extracting features from the border areas, e.g. by combining some pixels from the right edge of the first image with some pixels from the left edge of the second image, seems like a weird an unnatural thing for the classifier to do.

Alternatively, I could stack them in channels, thus creating one 600x200x3 image, where each channel constisted of one of the original images, padded with black pixels. This would probably make more sense in many ways, but still it seems slightly wasteful.

So I wonder: Is there a better way to combine them, and if so, how can this be implemented using the Matlab Neural Network toolbox? I was thinking something like creating a sub-network for each image, then at some point, the outputs of these sub-networks could be combined by a final network (with convolution layers, or simply a fully connected one with just a couple of layers).


texts:
I'm a beginner to the machine learning and data science. Somehow clear with the theoretical notions, but as a beginner i'm finding it a bit harder to implement. I've small query 'Whether the schema i'm trying to develop will work or not?'

I'm building a model to predict the emotional status using facial feature extraction. I've a model for that in the picture.

Actually I've got a set of images associated with the particular emotions. Basically i'm trying to perform the classification on the top. I'm detecting the edges of the eyes, mouth and performing regression over the pixel position for such edges.

My confusion: Whether the regression over the position would be efficient or not. If not what are the best ways to perform such task. I'm not concerned about neural nets or any other. I'm just trying to put it simple as possible.

Note: Please do help me to improve my questioning, so i can get a bit closer to what i'm asking. As a beginner i mightn't be able to express my query properly.


texts: Defining input shape in TensorFlow layers

texts:
I am trying to code my first TF program using Fashion-mnist data. In doing so i have some doubts regarding the usage of parameters while creating the layers. Irrespective of me defining the first layer with input_shape  & unit, the code seems to work just fine.
-> Can someone please help me understand why these 2 parameters are required at all?
-> I read that the "units" of each layer will define the output shape (the shape of the tensor that is produced by the layer and that will be the input of the next layer). Does that mean in my "Usage 1", the data shape changes from 28*28 -> 10 -> 128 -> 1? This is tough to visualise. Can someone please explain this?
Usage 1
'''
    import tensorflow as tf
    print(tf.__version__)

    mnist = tf.keras.datasets.mnist

    (training_images, training_labels) ,  (test_images, test_labels) = mnist.load_data()

    training_images = training_images/255.0
    test_images = test_images/255.0

    model = tf.keras.models.Sequential([tf.keras.layers.Dense(units=10, input_shape=[28,28]),
                                        tf.keras.layers.Dense(128, activation=tf.nn.relu),
                                        tf.keras.layers.Dense(1, activation=tf.nn.softmax)])

    model.compile(optimizer = 'adam',
                  loss = 'sparse_categorical_crossentropy')

    model.fit(training_images, training_labels, epochs=5)

    model.evaluate(test_images, test_labels)

    classifications = model.predict(test_images)

    print(classifications[0])
    print(test_labels[0])

'''
Usage 2
    model = tf.keras.models.Sequential([
                                        tf.keras.layers.Dense(128, activation=tf.nn.relu),
                                        tf.keras.layers.Dense(1, activation=tf.nn.softmax)])


texts:
I am reading about Inductive and Transductive Learning.

Some of the questions that come to mind are the following:

What is the difference between these two?
Which algorithms are usually employed for these methods?
Why would someone choose the latter over the former?
What are the best Python packages that implement these methods?

I have some answers to these questions, but I prefer to survey information from other people who have worked on semi-supervised learning problems for some time.


texts: How to apply feature selection in cross validated bagging

texts: Normally in cross validation decision tree, feature selection will occur with training data but in bagging ensemble the training data is bootstrapped. How can I apply feature selection in cross validate (independent test) bagging/ random forest without having bias? I dont want to use oob error estimation since I want to know recall and f1 measure. Anyone know?


texts:
I'm following along the NLTK book and would like to change the size of the axes in a lexical dispersion plot:

import nltk
from nltk.corpus import inaugural
cfd = nltk.ConditionalFreqDist(
            (target, fileid[:4]) # "[:4]" slices only the years of the speeches
            for fileid in inaugural.fileids()
            for word in inaugural.words(fileid)
            for target in ["liberty", "equality", "brotherhood"]
            if word.lower().startswith(target))

Because it gets very crowded otherwise:

cfd.plot(title="French ideals in US-American speeches through time")

the __doc__ doesn't seem to mention it:

print(cfd.plot.__doc__)

        Plot the given samples from the conditional frequency distribution.
        For a cumulative plot, specify cumulative=True.
        (Requires Matplotlib to be installed.)

        :param samples: The samples to plot
        :type samples: list
        :param title: The title for the graph
        :type title: str
        :param conditions: The conditions to plot (default is all)
        :type conditions: list

And I think there is nothing on it in the NLTK documentation, but also not in the matplotlib documentation (where I figured the plot functionality comes from):

http://www.nltk.org/py-modindex.html#cap-p
http://matplotlib.org/api/pyplot_api.html#matplotlib.pyplot.plot

(sorry for the code-block links, I'm new here and can't post more than 2 links but still felt like making it easier for someone reading this :)

I'd be glad if someone could point me into the right direction! Thanks!


texts: Best metric in imbalanced classification for multi-label classification

texts:
My test data are imbalanced, i tried to use the precision or the gmean as metrics for a multi-label learning model, but both metrics are not very informative. Is there any way to use for example the precision that gives information about the minor class and the accuracy in a single procedure. Any other solution for this problem is welcomed.

Thanks in advance.


texts:
I have a data set on predicting solar power generation,the dataset has 20 independent var and 1 dependent. The accuracy of my model is stuck at 60%. I have tried several models but this accuracy is best I could get other sucks even more.
Here is my code:
data_path = r'drive/My Drive/Proj/S.P.F./solarpowergeneration.csv'
dts = pd.read_csv('solarpowergeneration.csv')
dts.head()
X = dts.iloc[:, :-1].values
y = dts.iloc[:, -1].values
print(X.shape, y.shape)
y = np.reshape(y, (-1,1))
y.shape
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.25, random_state=42)
from sklearn.preprocessing import StandardScaler
sc= StandardScaler()
X_train = sc.fit_transform(X_train)
X_test = sc.transform(X_test)
y_train = sc.fit_transform(y_train)
y_test = sc.transform(y_test)

import keras.backend as K
def calc_accu(y_true, y_pred):
  return K.mean(K.equal(K.round(y_true), K.round(y_pred)))

def get_spfnet():
  ann = tf.keras.models.Sequential()

  ann.add(Dense(X_train.shape[1], activation='relu'))
# ann.add(BatchNormalization())
  ann.add(Dropout(0.3))
  ann.add(Dense(32, activation='relu', kernel_regularizer=regularizers.l2(0.01)))
 # ann.add(BatchNormalization())
  ann.add(Dropout(0.3))
  ann.add(Dense(32, activation='relu', kernel_regularizer=regularizers.l2(0.01)))
 # ann.add(BatchNormalization())
  ann.add(Dropout(0.3))

  ann.add(Dense(1))
  ann.compile(loss='mse',
              optimizer='adam',
              metrics=[tf.keras.metrics.RootMeanSquaredError(), calc_accu])
  return ann

spfnet = get_spfnet()
#spfnet.summary()
hist = spfnet.fit(X_train, y_train, batch_size=32, epochs=250, verbose=2)

the accuracy and loss graphs are
plt.plot(hist.history['calc_accu'])
plt.title('Model Accuracy')
plt.xlabel('Epochs')
plt.ylabel('Accuracy')
plt.show()
plt.plot(hist.history['root_mean_squared_error'])
plt.title('Model error')
plt.xlabel('Epochs')
plt.ylabel('error')
plt.show()

after 50 epochs nothing seems to improve, neither curve seems to overfit on the data
I tried other models like reducing layers and removing kernel regularizes, using
kernel_initlizers='normal' and 'he-normal'

but they perform poorly stuck at 20%.
dataset:


texts: Inputting (a lot of )data into a dataframe one row at a time

texts:
I'm using python. Some 2D numpy arrays are stored in individual rows of a Series. They are 30x30 images. It looks something like this:

     pixels
0    [[23,4,54...],[54,6,7...],[........]]
1    [[65,54,255,55,...],[43,54,6...],[......]]
...
...
...
7000 [[........]]

For each row in the Series, I want to take these 2D arrays, flatten them to 1D, take the values and assign them to the columns of one row in data frame. Each row will have 30x30 = 900pixels each, storing the values of each pixel. Like this:

    pixel1    pixel2    pixel3...    pixel900
0       23         4        54             77
1       65        54       255             33
...
...
...

I'm using an elaborate function that extracts one row from the series at a time, flattens the array, converts it to a Series again, and appends it to a dataframe. It takes sooo long. I'm sure there must be a faster way. I'm using this code:

def prep_pixels(X):
    # X is a series
    df = pd.DataFrame()
    for i in range(len(X.index)): #iterate through the whole series
        df = df.append(pd.Series(X[i].flatten()), ignore_index=True)
    return df

EDIT:
Upon request from a user, I will provide code with how I ended up in this rut in the first place :D

#reading the files
filepath = 'dataset.pickle'
data_np = pd.read_pickle(filepath)
print(data_np[0])

Output:

 [array([[255, 248, 253, 255, 251, 253, 254, 236, 220, 217, 191, 145, 139,
        185, 216, 227, 252, 251, 254, 248, 251, 236, 221, 222, 213, 175,
        120,  75,  74, 209],
       [255, 253, 254, 253, 252, 254, 223, 146,  87,  75,  58,  30,  27,
         58,  86, 116, 157, 168, 164, 165, 167, 136,  96,  71,  59,  49,
         21,   9,  27, 144],
       [255, 255, 255, 248, 252, 255, 202,  88,  15,  16,  14,  11,  11,
         12,  12,  20,  40,  46,  38,  43,  40,  25,  21,  19,  17,  35,
         53,  58,  64, 124],
    ... 30 rows of 30 pixels
    ...
    ... last row coming up ...
    [255, 255, 254, 254, 253, 252, 253, 254, 255, 255, 254, 252, 249,
    249, 251, 213, 126, 178, 231, 252, 248, 250, 254, 254, 252, 253,
    255, 255, 255, 255]], dtype=uint8), 'क']

The last symbol in this list is the character that this image represents. It's the 'label'. It's supervised learning using CNNs. Anyway, I need them to be in the other format I described to be able to work with them. This is how I'm handling this data:

data = pd.DataFrame(data_np, columns=['pixels','labels'])
def prep_pixels(X):
    df = pd.DataFrame()
    for i in range(len(X.index)): #iterate through whole series
        df = df.append(pd.Series(X[i].ravel()), ignore_index=True)
    return df

X = prep_pixels(data['pixels'])
y = data['labels']

EDIT: a user suggested that I use a mutable datatype to do this procedure. They said that it might speed things up because the computation does not need to make copies of data. I used some nested for loops and it cut the time to half (1 min 22 sec instead of 3 min). I still feel like its pathetic, given that my dataset has just 7000, 30x30 pixel images. Or, maybe I'm just new to data wrangling.

Here is the code I used. Please let me know if you have any other suggestions:

filepath = 'dataset.pickle'
data_np = pd.read_pickle(filepath)

df = pd.DataFrame()
for row in range(IMG_ROW):
    for col in range(IMG_COL):
        pixel=[]
        for img in range(len(data_np)):
            pixel.append(data_np[img][0][row][col])
        columns = pd.Series(pixel, name=col)
        df = pd.concat([df, columns], ignore_index=True, axis=1)


texts: Is there a common relationship between data inputs and the number of attainable features?

texts:
Is there a known relationship between the amount of information gain that comes from new data added to a dataset?
for eg: If I have a plant watering system that tells me:

An integer of  how wet the ground is (like a moisture sensor)
and 2) a regular measurement time

I have a new 'feature' 3) The change in moisture content over time.
If I start recording 4) How much water is added for each time interval,
I now have 5) How much water is being consumed (in some way).
If I add 6) ambient temperature information, I will start to see 7) water consumed by soil/plant vs 8) evaporated.
at this point I can start to deduct 9) an evaporation coefficient relative to ambient temp, and from that, I can start to generate 10) how much water the plant uses per day but also things like 11) how much water can keep the ground cool enough to reduce evaporation (perhaps).
So at some point, the amount of inferred information becomes greater than the number of inputs supplied.
Is there some known formula to understand how much information gain adding new data supplies, regardless of its eventual relevance?


texts:
I am currently attempting to output some data based on user agent strings. The strings are too messy to place into a plot similar to the one shown below, so I plotted their index instead. Why is my plot outputting out of order? I would like to have my $y$-axis be in ascending order.

For context, I am plotting 6 plots with a for loop and none of them are producing the y-axis in order.

Edit: I added 'uas_id' dplyr code with solution.

gbuas_large<-gbuas_large%>%
              mutate(uas_id = sort(strtoi(rownames(gbuas_large))))%>%
              select(user_agent_string,uas_id,everything())

col<-names(gbuas_focus[,5:10])

for(i in col) {
  print(i)
  ua_p<-ggplot(gbuas_focus
              )+
    geom_point(aes_string(
                x=i,
                y='uas_id'),
                color="#70a6ff",
                size=3
               )+
    labs(x='Variable of Score',
         y='User Agent Strings',
         title="Understanding Misclassifying Variables"
        )+
    facet_wrap(~classification)
    print(ua_p)
}


texts: How is the E(X) of a Poisson distribution lambda?

texts:
I was recently learning about the Poisson distribution, and was very perplexed about the E(X) equaling lambda. Like how did lambda even come into the picture here, isn't it a symbol of wavelength?
Also isn't wavelength just a smaller unit of distance? In that case isn't saying E(X)=5 cars per hour(assume) wrong? Because Distance=Speed * Time, and saying per hour signifies division.

Thanks in advance for solving my doubts. If possible please explain everything in layman's terms.


texts: What is word embedding and character embedding ? Why words are represented in vector with huge size?

texts:
In NLP word embedding represent word as number but after reading many blog i found that word are represent as vectors ? so what is word embedding exactly and  Why words are represented in vector and the vector size is huge. what values dose that vector represent ?

what is the difference between word and character embedding ? please explain in simple term with simple example.


texts: Finding the inner product of two 3D tensors in Keras custom layer

texts:
In my Keras custom layer, I have to find the inner product of two 3D tensors. For example , x=(?,80,150,12) is the input to the layer and the inner product have to be with all the 4 slices of kernel (80,150,12,4), 4 being the slicing dimension. I am trying to do it with K.dot but I am getting dimension mismatch error. The code snippet is

kernel1_unpacked = tf.unstack(self.kernel1,axis=3)
 all_probs = []

 for t in kernel1_unpacked:
     t=K.permute_dimensions(t,[2,0,1])
     B3=K.sum(K.dot(x,t),[1,2,3])
     B3=K.expand_dims(B3,1)
     print(B3)
     all_probs.append(B3)

 B4=tf.concat(all_probs[:],axis=1)

 print(B4)

the output should have dimension (?,4).


texts: In CS231n lecture, can't the linear classifier be softmax itself?

texts:
I am little bit confused on why the scoring function that is the $f(X,W)$ is chosen to be $W,X$ while they talk about Softmax and SVM loss in this.

Can't they take Softmax classifier or SVM classifier and then explain the losses?

Was there a particular need of taking the above mentioned scoring function?


texts: Can I replace an embedding layer with hot encoding layer?

texts:
I'd like to build an autoencoder for a movie recommendation. My data looks like below
user, movie, rating
1, 123, 3
1, 333, 5
2, 34, 3
...

I have seen people using embedding layer as the first layer of their network; two embedding layer, one for users and one for movies. I was wondering if I can remove that embedding layer if I apply hot-encoding on my data and convert it to below format
 user1, movie1, movie2, movie3, ...., movie_n
  1   , 3, 4,5, ...
  2,  4, 4, 5, ...


texts:
I have 50000 observations with 70% positive and 30% negative target variable. I'm getting accuracy of around 96-99% which seems unreal of course and I'm worried that my model is over-fitting which I don't understand why. I replaced all outliers with 5th and 95th quantile. Standardized the data yet it is showing this unreal accuracy.
A bit of online search and people suggested to check for the difference between training and test data accuracy, which i did and for Random Forest it came

Training Accuracy: 0.997975
Test Accuracy: 0.9715

For logistic regression it shows

Training Accuracy: 0.967225
Test Accuracy: 0.9647

This is the code I used for running the model:

clf = LogisticRegression()
trained_model = clf.fit(X_train, y_train)
trained_model.fit(X_train, y_train)
predictions = trained_model.predict(X_test)

accuracy_score(y_train, trained_model.predict(X_train))
accuracy_score(y_test, predictions)

I also tried kfold cross validation which gave similar results

skfold = StratifiedKFold(n_splits=10, random_state=100)
model_skfold = LogisticRegression()
results_skfold = model_selection.cross_val_score(model_skfold, X, Y, cv=skfold)
print("Accuracy: %.2f%%" % (results_skfold.mean()*100.0))

Lastly I applied regularization technique to check for results and this is the result  I got

for c in C:
    clf = LogisticRegression(penalty='l1', C=c, solver='liblinear')
    clf.fit(X_train, y_train)
    y_pred_log_reg = clf.predict(X_test)
    acc_log_reg = round( clf.score(X_train, y_train) * 100, 2)
    print (str(acc_log_reg) + ' percent')
    print('C:', c)
    print('Coefficient of each feature:', clf.coef_)
    print('Training accuracy:', clf.score(X_train_std, y_train))
    print('Test accuracy:', clf.score(X_test_std, y_test))
    print('')

The results

96.72 percent
C: 10
Coefficient of each feature: [[-2.50e+00 -1.40e-03  2.65e+00  4.09e-02 -2.03e-03  2.75e-04  1.79e-02
  -2.13e-03 -2.18e-03  2.90e-03  2.69e-03 -4.93e+00 -4.89e+00 -4.88e+00
  -3.27e+00 -3.30e+00]]
Training accuracy: 0.5062
Test accuracy: 0.5027

96.72 percent
C: 1
Coefficient of each feature: [[-2.50e+00 -1.41e-03  2.66e+00  4.10e-02 -2.04e-03  2.39e-04  1.68e-02
  -3.29e-03 -3.80e-03  2.52e-03  2.62e-03 -4.22e-02 -9.55e-03  0.00e+00
  -1.73e+00 -1.77e+00]]
Training accuracy: 0.482525
Test accuracy: 0.4738

96.74 percent
C: 0.1
Coefficient of each feature: [[-2.46e+00 -1.38e-03  2.58e+00  4.03e-02 -1.99e-03  2.22e-04  1.44e-02
  -4.49e-03 -5.13e-03  2.03e-03  2.20e-03  0.00e+00  0.00e+00  0.00e+00
   0.00e+00 -6.54e-03]]
Training accuracy: 0.616675
Test accuracy: 0.6171

95.92 percent
C: 0.001
Coefficient of each feature: [[-1.43e+00 -6.82e-04  1.19e+00  2.73e-02 -1.10e-03  1.22e-04  0.00e+00
  -2.74e-03 -2.55e-03  0.00e+00  0.00e+00  0.00e+00  0.00e+00  0.00e+00
   0.00e+00  0.00e+00]]
Training accuracy: 0.655075
Test accuracy: 0.6565

The codes I used for Standardization and replace outliers

std_scale = preprocessing.StandardScaler().fit(X_train)
X_train_std = std_scale.transform(X_train)
X_test_std  = std_scale.transform(X_test)

X.clip(lower=X.quantile(0.05), upper=X.quantile(0.95), axis = 1, inplace = True)

Do let me know if any other information is required and any guidance will be appreciated


texts:
I have ~100 sets of samples with integer IDs. For example, 3 of them could be:

a = [0, 1, 3, 4, 6...]
b = [1, 5, 9, 102...]
c = [1, 7, 10, 42...]

I am looking to cluster/group together these sets such that within each cluster, all the elements have at least X% common IDs with each other, where X is an input parameter.

I was thinking about using Agglomerative Clustering with 1 - %X as the distance metric, but was unsure how to modify it to account for each clusters 'information' being the common set of IDs between sets within it.
Any advice would be appreciated (including a different technique/algorithm clustering was just what came to mind)


texts: Does click frequency account for relevance?

texts: While building a rank, say for a search engine, or a recommendation system, is it valid to rely on click frequency to determine the relevance of an entry?


texts:
Assume a set of loosely structured data (e.g. Web tables/Linked Open Data), composed of many data sources. There is no common schema followed by the data and each source can use synonym attributes to describe the values (e.g. "nationality" vs "bornIn").

My goal is to find some "important" attributes that somehow "define" the entities that they describe. So, when I find the same value for such an attribute, I will know that the two descriptions are most likely about the same entity (e.g. the same person).

For example, the attribute "lastName" is more discriminative than the attribute "nationality".

How could I (statistically) find such attributes that are more important than others?

A naive solution would be to take the average IDF of the values of each attribute and make this the "importance" factor of the attribute. A similar approach would be to count how many distinct values appear for each attribute.

I have seen the term feature, or attribute selection in machine learning, but I don't want to discard the remaining attributes, I just want to put higher weights to the most important ones.


texts: How do decision tree works for feature selection?

texts:
I have a dataframe with a feature selection problem. I want to get the variables explaining the variance within each segment of the following dataset:

    Do you agree    Gender  Age     ...  City          Urban/Rural  Output
0   Yes             Female  25-34   ...  Madrid        Urban        Will buy
1   No              Male    18-25   ...  FÃ¨s-MeknÃ¨s  Rural        Won't
2   ...             ...     ...     ...  ...      ...               Undecided
....

The target being Output.

I've been told a decision tree could be a way so after googling a bit I did:

# Feature Importance with Extra Trees Classifier
from sklearn.ensemble import ExtraTreesClassifier

# feature extraction
model = ExtraTreesClassifier(n_estimators=10)
model.fit(X, y)
print(model.feature_importances_)

Which returns:

[0.         0.00473011 0.00716472 0.00778101 0.0051573  0.00139121
 0.02045262 0.00791912 0.         0.00222593 0.00173901 0.00417362
 0.00222593 0.00313295 0.00565095 0.00652543 0.         0.00527774
 0.         0.         0.00601354 0.         0.         0. ...

This looks exactly what I am looking for, but I don't get how we obtained this output. It look a bit like a black box. I am eager to learn and to do it myself to understand.Can you explain to me or refer me some resoures that would help me implement it myself?

Annex: data preparation

def load_dataset():
    connection = psycopg2.connect(user = "user",
                                  password = "password",
                                  host = "host",
                                  port = "5432",
                                  database = "database")
    connection.set_client_encoding('UTF8')
#     connection.set_client_encoding('UNICODE')
    sql = "select * from capi limit 10;"
    # load the table
    df = pd.read_sql_query(sql, connection)
    # retrieve numpy array
    dataset = df.values

    # split into input (X) and output (y) variables
    filtered_cols = ['Output']
    cols = [col for col in cols if col not in filtered_cols]

    X = df.loc[:, cols]  #independent columns
    X = X.astype(str)
    y = df['Output']    #target column i.e price range
    return X.values, y.values

X,y = load_dataset()


texts:
I have a questionnaire for different years. The questionnaire layout is the same for every year. The questionnaire has Likert-scale definitely agree to definitely disagree. The questions are the in one column and the number of responses is corresponding with the liker scale, which is in each row from top to bottom.

|-------------| Category1| Q1 | q2 |  Category2| Q1 | q2 |  q3
|-------------| Average  |    |    |  Average
| agree       | 5        | 5  | 5  |      8    | 9  | 6  |  1
| disagree    | 4        | 2  | 2  |      3      4    3     3

This an example of the data.

Would I use the survey package? what benefits are there? Or should I use a normal association rule and have an association between the question? This is just an example to see any trends and patterns.


texts: A cross-entropy loss explanation in simple words

texts:
Suppose I build a FNN model. The last layer is a classification layer with softmax activation.
A cross-entropy loss is used to classify a problems, such as logistic regression.
How would I calculate the cross entropy loss ?

But I can not fully understand the cross-entropy loss Please any convenient explanation

Thanks in advance


texts:
i have a data set with personal properties (age, education etc.), and each entry in the dataset is associated with a list of websites and daily visit.

my goal is to predict for a given person the list of websites he will visit, and to predict, for each website, how many daily visit it will have from that person.

the number of websites each person visits is very limited, usually between 3 and 20 (same websites for the whole dataset).
the number of different persons is limited (a few hundreds) but the number of entries per person is large (a few thousands, each representing a day).

currently my thoughts are to have two stage method -
first stage is for each possible website a person might visit (like mentioned before, usually 3-20), classify that website as "visited" or "not visited" (using binary classifiers for each website).

stage two is for each website that was classified as "visited", use a linear regression to predict the number of daily visits.

I am very new to the field of ML so dont spare your words! your comments and tips are very appreciated :)
Thank you!


texts:
I am searching a scientific database for abstracts of papers containing the words project management. Here is the link:

For getting abstracts, I need to click on any paper and open a new page. How can I do that for 68 papers? I program in R and bash.


texts: Having trouble understanding the x and y axis in SVM when training and testing data

texts:
I wrote some code based on this article.

In the code in the article they have created a partition of 80 percent test and 20 percent data

#What percentage of data you want to keep for training
percentage = 80
partition = int(len(hog_features)*percentage/100)

Later they have created the following variables, the data frame is two dimensions np array --- data_frame[hog_features,labels]:

x_train, x_test = data_frame[:partition,:-1],  data_frame[partition:,:-1]
y_train, y_test = data_frame[:partition,-1:].ravel() , data_frame[partition:,-1:].ravel()

clf.fit(x_train,y_train)

now what I don't understand is why for the second dimension of the array (the labels) for the x variables they're including everything except the last value and for the y variables the second dimension of the array is just including the last element of the array from what I can understand using array slicing (Maybe I'm wrong).


texts: What layer to consider for embeddings when building a encoder-encoder model?

texts:
I am currently building a encoder-encoder based model with cosine loss function. The dataset is supervised learning.
Here's a basic encoder,
from tensorflow.keras import layers

class LSTMEncoder(layers.Layer):

    def __init__(self,
                 units,  # dimensionality of the output space
                 input_dim,  # vocab of size
                 output_dim,  # embedding dimension
                 name='encoder',
                 **kwargs):
        super(Encoder, self).__init__(name=name, **kwargs)
        self.embedding = layers.Embedding(input_dim=input_dim, output_dim=output_dim)
        self.lstm = layers.LSTM(units=units)

    def call(self, inputs):
        emb = self.embedding(inputs)
        return self.lstm(emb)

Both the encoders are very similar, and has a output layer since it's a supervised learning,
I added a dense layer at the end for probabilities and loss func is consine loss
Now, My goal is not to predict the classes, rather getting the trained embedding for each text sentence from encoder, but encoder has both embedding layer and lstm layer
so I want to know whether to cosider the output of lstm layer as trained embeddings or should I extract embeddings from embedding layer?


texts: How can I save a final model after training it on chunks of data?

texts:
After training a model on chunks, how can I save the final model?

df = pd.read_csv(, chunksize=10000)
for chunk in df:
  text_clf.fit(X_train, y_train)
  filename = 'finalized_model.sav'
  joblib.dump(text_clf, filename)

# load the model from disk
loaded_model = joblib.load(filename)

Saving a model like this will just give me the model trained on the last chunk. How can I avoid that and get the overall model trained on every chunk?

UPDATE:
 Most of the real-world data sets are huge and can’t be trained in one go. How can I save a model after training it on each chunk of data?

df = pd.read_csv(“an.csv”, chunksize=6953)
for chunk in df:
  text = chunk[‘body’]
  label = chunk[‘user_id’]

  X_train, X_test, y_train, y_test = train_test_split(text, label, test_size=0.3 )

  text_clf = Pipeline([(‘vect’, TfidfVectorizer()),
  (‘tfidf’, TfidfTransformer()),
  (‘clf’, LinearSVC()),
  ])

  text_clf.fit(X_train, y_train)

  # save the model to disk
  filename = ‘finalized_model.sav’
  joblib.dump(model, filename)

Will saving it this way give me the model trained on the entire dataset? I want the model trained on every chunk. Any help?


texts: Find Net Reclassification Improvement/Index metric using Python

texts:
I am working on a binary classification problem with ~5k records and class proportion of 33:67.

I have 60 features/variables in my dataset and finally I have come to about 10 variables based on multiple feature selection algorithms and domain understanding.

Now I would like to experiment with this 10 features and get the best of these 10 which gives the high performance.

For example, Having 6 features might give an AUC of 84% whereas adding one new feature (7 features) might give an AUC of 85%.

But adding two features (8 features), might give an AUC of 83%.

I read in several posts and F Harrel blog posts that AUC is not always a good metric to compare model performance.

q1) So what are the other best metrics that you usually use?

q2) How to calculate Net Reclassification Improvement? Read it's better than AUC.

Can you people please help me?


texts: Working with audio data with different sample rates in Tensorflow

texts:
I am trying to implement (as a toy project) some aspects of speech recognition in Tensorflow. The audio files I want to use as training and test data have different sample rates (16, 20, 44 and 44.1 kHz).

The following list is not exhaustive, just listing a few options I could think of.

I can resample the whole signal (which is quite un-periodic) using scipy.signal.resample, probably after padding with zeros to achieve a signal length which is a power of 2, chopping of the equivalent amount afterwards.
I can try to chop the signal into short, non-overlapping windows, which I modify using scipy.signal.resample before glueing them back together and applying the rest of the chain.
I can chop the signal up into overlapping windows, scipy.signal.resample them and stick them together in a weighted fashion.
I can take the spectrogram tensor obtained using tf.contrib.signal.stft and try to stretch it, for example using rational function interpolation (reflecting tube models of the vocal tract)
I can assume that the model will learn to abstract away from sample rates, maybe even helping it (or hindering?) by adding convolution layers before the actual model I am interested in.

In which step of the model should I normalize sampling rates, and how should I do it?


texts:
Can I do online learning with random forests? I have a few million datapoints and the classifier fails to finish on the cross validation step.

Can i break it up in chunks sequentially?
Current code:

X_train, y_train, X_val, y_val, X_test, y_test = load_dataset()

   print('Planting trees...')
   clf = RandomForestClassifier(
       n_estimators=50,
       max_depth=None,
       min_samples_split=1,
       random_state=0
   )

   print('Growing trees...')
   classifier = clf.fit(X_train, y_train)

   # see how we did
   print('Testing trees...')
   scores = cross_val_score(classifier, X_test, y_test)
   print(scores)
   print('accuracy: %d' % (scores.mean()))

Can I change it to something like:

for chunk in df:
     clf.fit(...)
     cross_validate...


texts: How do I get my Neural network to ignore certain values?

texts:
I was wondering if there was a way that I can get my CNN encoder-decoder neural network to completely ignore certain values in my data (2d images).
There are some pixel values of 0 that never change from the input to the target. They are meant to be ignored. I've tried some self attention layers and I even multiply the 'mask' image at the end of the encoder-decoder network, but it still does not ignore these values, causing a lot of unnecessary noise and inaccuracy in my results.
How can I ensure these values are being ignored at every convolutional layer?


texts: Continuously train CoreML model after deploy on Server?

texts: I am working on face recognition with CoreML.I make a CoreML model with python Turi Create and put it on Server. User download it and compile on device but there is one problem if we want to do any updation like add a new user's face then we have to make this model again and then put it on Server.I want that if i deploy a model on Server then for every new user i train this model on the server.Is there any possible ways to do this.


texts: Is there "Attention Is All You Need" implementation in Keras?

texts:
Has anyone seen this model's implementation using Keras?

inb4: tensorflow, pytorch


texts: Predicting future order dates and order amounts for customers with online retail data

texts:
Using the online retail II dataset (https://archive.ics.uci.edu/ml/datasets/Online+Retail+II), I'm trying to predict when each customer will place subsequent orders, and if possible, the monetary value of those orders.  This will be a proof of concept for a real online retail store, in which I'll have access to even more data/features.

This current dataset contains the following attributes:

InvoiceNo: Invoice number. Nominal. A 6-digit integral number
uniquely assigned to each transaction. If this code starts with the
letter 'c', it indicates a cancellation.
StockCode: Product (item) code. Nominal. A 5-digit integral number
uniquely assigned to each distinct product.
Description: Product (item) name. Nominal.
Quantity: The quantities of each product (item) per transaction.
Numeric.
InvoiceDate: Invoice date and time. Numeric. The day and time when a
transaction was generated.
UnitPrice: Unit price. Numeric. Product price per unit in sterling.
CustomerID: Customer number. Nominal. A 5-digit integral number
uniquely assigned to each customer.
Country: Country name. Nominal. The name of the country where a
customer resides.

My current intuition is to model the data as a daily time series that would output which customers are likely to order for the day and the value of those orders (if possible?).  The output would either be a sparse, one-hot encoded matrix of shape num_customers X num_customers, or a more dense vector of size num_customers (which could have thousands of customers).  I'm not quite sure how, or even if the model would be able to also output the amount of the order.  This model will likely be fed into an LSTM network or similar. I suppose this would be considered a multivariate time series forecast (still learning here)?

From this, I'm hoping to be able to run the prediction for each day in the following year to forecast the next order date, customer value and churn risk

Since this would be an attempt to model human behavior as a time series, without a direct correlation from one day to the next (in contrast to predicting daily temperature or a stock price), and with so many variables, I'm also worried it would just come off as unpredictable noise.

Anyway, am I on the right track? (any insights would be greatly appreciated)
Should this be broken into multiple models, and if so, how many?
Does anyone have any tips, or know good articles or tutorials that could point me in the right direction?
What are the merits of the one-hot encoded matrix, vs the multi-hot
encoded vector (not sure if that's the proper name)?

Thanks!


texts:
I have a sorted dataset by datestamp which looks like this:

user    message
A       Hi.
B       Hello.
B       How are you?
A       I am stuck.
B       How can I help you?

What I want is to create a pandas df that would look like this:

user  message       reply
A     Hi.           Hello.
A     Hi.           How are you?
B     Hello.        I am stuck.
B     How are you?  I am stuck.
A     I am stuck.   How can I help you?

For each message, I want to find all of the replies. That means that I want the messages after current one but from the other user. How can I do this with pandas? Let's only consider a binary case of 2 users A and B.


texts: Python package for machine-learning aided data labelling

texts:
In a lot of cases unlabelled data needs to be transformed to labelled data. The best solution is to use (multiple) human classifiers. However, going to all the data by hand (i.e. in text-mining or image-processing) is often a daunting task. Is there software that can combine human classifiers and machine-learning techniques in real time? I am especially interested in python packages.

To illustrate, classifying images from video streams is very repetitive. After 100 images (from different streams) a machine-learning algorithm could be used to predict the labels given by the human classifier. The machine classifier might be very confident about some (un)seen samples and very uncertain about others. The human classifier can then focus on the uncertain samples helping the machine classifier to learn better what is does not yet know.


texts: PCA or cluster table of experimental fitness scores

texts:
I need to find patterns experimental data.

The columns are "experiments" which are chemical treatments for growth experiments.  The rows are individual gene names, the values are a fitness-defect score, which reflect the genes contribution to growth.

I would like to find patterns that are reflected across all experiments using some type of PCA or clustering.  I have been trying to use sklearn but have not been successful in applying a model.

The data looks like:

gene     SGTC_1                   SGTC_2                 SGTC_3
YAL002W  3.56420220283773        1.80774301690328       0.431491057210906
YAL004W -0.885645399324204      -1.76020417788351       0.883034190306176

....

There are 4000 rows for genes and 30 columns for experiments.

Any suggestions would be greatly appreciated.


texts: Maximum number of images to train a convolutional neural network

texts:
I just wondered if there is a technical limit on the number of images to train a neural network.

I want to work with extremely high numbers of images, around 1,000,000 to 10,000,000 images. Is there a cap because of the graphics card memory? I tried to do some research on that issue, but I mostly find questions about how many images you need to get decent results for a convolutional neural network.

Thanks and best regards,

beinando


texts:
While running the below pig script I am getting error in line4: If it is GROUP then I am getting error. If I change from GROUP TO group in line4, then the script is running. What is the difference between group and GROUP.

LINES = LOAD '/user/cloudera/datapeople.csv' USING PigStorage(',') AS ( firstname:chararray, lastname:chararray, address:chararray, city:chararray, state:chararray, zip:chararray );

WORDS = FOREACH LINES GENERATE FLATTEN(TOKENIZE(zip)) AS ZIPS;

WORDSGROUPED = GROUP WORDS BY ZIPS;

WORDBYCOUNT = FOREACH WORDSGROUPED GENERATE GROUP AS ZIPS, COUNT(WORDS);

WORDSSORT = ORDER WORDBYCOUNT BY $1 DESC;

DUMP WORDSSORT;


texts: Macro F1 result higher than accuracy for imbalanced dataset

texts:
In one of the research papers on fake news detection, the authors collected a fake news binary dataset (fake vs. real news) consists of 16,817 real articles and 5,323 fake ones.

The authors presented the results using accuracy, precision, recall and F1 without specifying which kind of averaging they applied on the F1 metric (macro, micro, weighted, etc.).

Here are the results:

If you can notice for the last system, the accuracy value is 0.689 and the F1 value is 0.717 which is higher than the accuracy.

Thus, give the imbalanced status of the dataset, is it possible that the authors averaged the classes in the F1 metric using macro way?

For me, this "impossible" to happen, and I argue that probably they used weighed F1 score.


texts:
The orange curve is train accuracy and blue is validation accuracy. Is this clear overfitting or should I let it run for more epochs?

With custom dataset (1D data with 70 features) I trained a 2 layer MLP.
Network Architecture: [70-200-200-4].
I'm only able to reach ~50+% accuracy. Any suggestions on what steps I can take to improve accuracy? (Obtaining more data isn't an option)

Thanks in advance!


texts: Determine when entry/series of entries are outliers

texts:
A common question I face is this:

I have a stream of incoming data. Let's call it a vector of entries where each entry represents a value. As I'm getting this stream of entries that is getting added to the vector, I want to figure out if one of them is an outlier. So my question is two essentially:

How do you establish a good baseline as you consistently keep getting data?
How do you determine if a value that is appendede to the vector is significantly different than the baseline?


texts: How to determine number of Neurons setup in Convolutional Neural Networks?

texts:
Im new to deep learning and still learning on how to train my neural networks from the scratch. Sometimes I watch tutorial on YouTube or even online courses on the MOOC platforms. The base of a Convolutional Neural Networks usually has Conv2D and MaxPooling layers to make the input much more smaller and easy to be trained.

The thing is sometimes this tutorial online use the setup of Conv2D with higher number of neurons follows with other Conv2D with smaller one. Such as 1st layer is Conv2D(512...), 2nd layer is Conv2D(256....) and etcetera. Another tutorial setup is the increasing one, such as starts with Conv2D(16....), then Conv2D(32....) and increasing. These teacher doesn't tell us why they code the setup as so.

How do we know which setup to use?
Is there any differences between them? Can't find on the net how to refer this case.
If there's a paper that already described this case I would like to have it thanks.


texts: What is the current state-of-the-art video classification technique?

texts:
For a project I am aiming to automise the detection of goals in foosball (a.k.a. 'table football') matches. To do so I now track the ball in every frame using the openCV library in Python.

To determine if a ball has been shot into the goal and should count as a point I want to train a neural network on videos that include goals and non-goals. When the ball is not found for a number of frames, I want the algorithm to put the video through a neural network to determine if it disappeared because a team scored.

I've been doing some research on video classification techniques but cannot really determine which one works best on classifying videos right now.

Currently I am considering first training a CNN on single frames out of the videos, and then gathering the convolutional features for the videos by feeding them through the network (with classification layer and fully-connected layers popped off), after which the convolutional features are put through an LSTM classification network sequentially for each video.

Is this the best way to go about such a problem, or are there better techniques available?


texts: How to analyse player and enemy position for data analysis

texts:
I am trying to analyse a Serious Game for students learning.

In one of the game levels,
There are multiple positions like Player Position (x, y, z), enemy position (x, y, z), Player shot at position (x, y, z).
Other features like items acquired,

How do I use these for analysis, for e.g. to categorize some students as "more-exploring" than others.

I am sorry if the question is not clear. But I am just confused with the project.


texts: unsupervised learning time series datasets

texts:
I experiment on building electricity power consumption datasets and try to see relationships of the power consumption with weather data and dummy variables that represent time-of-week.
The only thing I do at the moment is OLS regression with Python statsmodels but I am curious if anyone could recommend an unsupervised learning process that could potentially do something similar, that is if an unsurprised learning process can something similar.
This is just a screenshot below, sorry but I use OLS to see the relationships for example the coefficients are in units of building electricity kW where the intercept is the baseload electrical usage and what impacts weather (dewpoint temp in this case) and hours of the day have on building power consumptions.
Strange question, curious about any thoughts! Any unsupervised learning package to try with some sort of visualization of the data relationships for this type of use case would be greatly appreciated.


texts: Investigate why data is missing? After finding out reasons, what should I do next?

texts:
x1 x2 x3 x4 - - - - - - x10 .... x21 x22 x23
 1                                           |
 .         Complete                          |
 .           data                            |
 88                                          |
 89                                          |
 90 ------------------------------------------
 .          |              |
 .complete  |   missing    |  complete
 .          |              |
 .          |              |
 . -------------------------------------------
 100
 101           Complete data
 102------------------------------------------

The dataset looks like this. 10 % of the data is missing. It doesn't appear missing at random.

From row 90 to 99. Variables x4 to x10 are missing. All other rows do not have any missing values. It is not missing at random. Is there any statistical way to investigate why they are missing.

My initial plan is to create a new column, 0 is not missing, 1 is missing.
Run a logistic regression on non-missing columns. Is this the correct way to do it or not?

 My questions:
     How should I investigate why data is missing by just playing around with the data set?

     If I found out the reasons or data is not missing not random, What should I do next?


texts: Binary classification using images and an external dataset

texts: I currently have a project in which I must create a binary classifier to detect defective products. I have image data which has already been labeled (each part has been labeled as a pass or fail), as well as an external dataset which has specific measurements for each image (it contains parameters which could not be accurately captured by a camera, such as temperature, length of a particular piece, etc.). I have achieved decent accuracy by applying basic classification algorithms such as logistic regression and support vector machines (in scikit-learn) to only this external dataset, but I would like to incorporate the image data to improve accuracy. I have read about training a CNN on the images, but I do not know how to incorporate the external dataset to train the model (in fact, I don't even know if this is possible). Is there any way to use both the image data as well as the external dataset to classify these images? Any help is much appreciated.


texts: Predicting equipment failure with time series alarm data

texts:
I am trying to predict machine failures based on alarm data.
The situation:
There is approximately 4000 machine failures per year. These are labelled poorly (it is entered manually and can have multiple names for the same failure). This dataset consists of the timestamp, failure name and failure description in the following way.
timestamp, name, description
01/01/2020 - 08:10, Motor Failure, Motor failed due to overheating

Also there is approximately 1 million alarms per year showing possible issues (There are 8000 possible alarms in the system but only ~1200 makeup the 1 million that activate). Alarms fire at the point of failure, but also in a large percentage of cases there are multiple related alarms activating for weeks before hand that are unable to be acted upon due to the shear number. This dataset consists of the timestamp, alarm id, and alarm description like the following.
timestamp, alarm id, description
01/01/2020 - 08:10, MFHeatHiHi, Motor temperature critical

Please excuse my ignorance as I am new to data science. I am trying to work out the best way to clean up/modify the data first and then in which directions to go for creating the prediction. Please let me know if I am going about it the wrong way or have the incorrect idea on where to start.

Update the 4000 failures with the same name for same events (I estimate there are approximately 600 discrete failures within this). Or I call them all the same name "failure" and test the alarms against only one variable and use the alarm description to give me the issue that may cause the failure.

Break the alarms in to time windows proceeding the failure (possibly 30 days or less). Then use the alarm id - description as columns with the number of occurrences as the value.

For example:
Failure timestamp, MFHeatHiHi - Motor temperature critical, FanHeatHiHi - Fan temperature critical,
01/01/2020 - 08:10,                4,                                        3,

Then I was thinking of using random forest with R.
Is this the right way to go about it, should I be making failure just one variable, is removing the time component of the alarm and transforming to number of alarms the correct way to go about it and would that mean I need 8000 columns for every possible alarm.
I am pretty sure I have the wrong method so would really appreciate some guidance so I am heading in the right direction.
Thanks for your help,


texts: Does it make sense to use KNN to target ticket buyers for arts events?

texts:
I work for a medium-sized non-profit. We have a database with approximately 40,000 profiles of folks (mostly patrons). For each show a patron attends, we add an attribute that indicated that they're into that kind of show. For example, if they saw a folk music show, they would get a Folk Music tag.

Does it make sense to use some form of the K-nearest neighbors algorithm to figure out who we could target to purchase tickets? Maybe along with some location data?

For instance, we have about a 100 seat theater, so if we have sold out about half a show, would it be effective to use KNN to try to find other patrons similar to those 50, or is that too small a sample-size?


texts: Similarity matching between two distinct datasets (marketing case study)

texts:
I am working for a company that sells different products to customers. My objective is to find customers that are likely to purchase product X based on the profiles of customers that already purchased product X.
My first idea was:

to collect relevant variables for customers that already purchased product X (dataset A)
perform a cluster analysis of this dataset to generate customer personas for dataset A
collect the same variables for customers that have not purchased product X (dataset B)
and finally measure the distance between customers in dataset B to the centroids of medoids of the generated clusters of dataset A

Unfortunately, this is less straightforward than I thought:

For one, I would have to cluster categorical and numerical data. Therefore, I would compute the gower-distance to get a dissimilarity matrix between data points of dataset A that I would then cluster by means of PAM (partitioning around medoids) clustering. I do not know how to apply data points of dataset B to infer a distance to the PAM medoids because those medoids relate to the dissimilarity matrix of dataset A rather than the actual data points.
Secondly, the generated clusters of dataset A are less descriptive than expected.

In conclusion, I would like to have a second opinion. Is the way I described really a good way to tackle the problem at hand? Or do you have other ideas?
Would be happy for your input - best wishes.


texts:
I am quite new to machine learning and python as well. I faced an imbalanced dataset and wanna use cross validation and oversamopling like the figure shown.

I realised the Python function below cannot be directly used for this purpose and please advice some codes for this task.

cross_val_score(model, X_train,np.ravel(y_train), cv=n_folds, n_jobs=1, scoring='roc_auc')


texts:
I am trying to wrap my head around LSTM networks, specifically about the input dimensionality.

Suppose we have the following dataset

Timestamp   Value
1   32
2   44
3   52
4   33
5   23
6   11

And we derive a new dataset from this one that will have time lag of 3 timestamps for training our LSTM network:

Value (t-3) Value (t-2) Value (t-1) Value (t)
32  44  52  33
44  52  33  23
52  33  23  11

Value(t) being the target value.

Suppose that we want to construct an LSTM network with a single hidden layer, containing 5 LSTM cells, followed by a Dense layer (not really relevant for this question) and a single target output.

This is what we would normally prepare for Keras, and it would do the rest.

I am interested in how this architecture would be "drawn out" on a piece of paper.

Specifically, what is the shape of the input vector for each LSTM cell? I understand the shape of X is (# rows, # features, # timestamps), but I don't really understand how it is fed into the LSTM cells. Because of all the Back Propagation Through Time, and the connections between each LSTM cell with its previous timestamp, is each of the values fed one by one? For example, for each of the 5 hidden LSTM cells, we first feed it Value (t-3), then using memory from what it learnt there we feed it Value (t-2) and so on? Since we have time lag of 3, there should be 3 time-steps for each LSTM cells. But is the shape of the input vector for each LSTM cell 1?

Also, what about text processing? Suppose each sentence is treated as a sample in the dataset. Sure, we'd encode each word with word2vec, but then how would we feed it to the neural network (each sentence can have different number of words, so the overall input vector can vary)?

I am a novice learner, please excuse my stupidity. Have a nice day.


texts:
I have a question abut MySQL

I would like to handle a column with characters without changing the original DB. Below is the levels of the column named 'Maker'.

기아, 현대, BMW, VolksWagen, Porsche
So there are about 10 기아s, 20 현대s, 30 BMWs and so on.

As you can see, some of them are written in Korean and others are in English.
I do not want to touch English-written words. I only want to change Korean words into English.

I tried to you CASE and IF sentences like below.

SELECT
    CASE
        WHEN Maker='기아' THEN 'KIA'
        WHEN Maker='현대' THEN 'HyunDai'
    END AS new_Maker
FROM table_name

It successfully changes KIA and HyunDai but BMW, VolksWagen, and Porsche shows as NULL.
I would appreciate any advice. Thank you.


texts: What Unsupervised Machine Learning algorithms can be used to detect Fraud in insurance?

texts:
I'm about to start a project regarding Fraud Detection in Insurance but there is no dependant variable to train the model and classify it.

Please provide the algorithms that I can use to detect Fraud in both Auto and Health insurance industry.

Thanks in advance!


texts: How can I know the method that HMM use to choose the hidden states automatically?

texts:
As I know, I can just decide the number of hidden states, then put my observations fit into my HMM model, the model will generate the hidden states for me.
I really curious how it does work. Can anyone explain the calculation or share the link which has such details to me.
Thanks!


texts: Create most "average" cosine similarity observation

texts:
For a recommendation system I'm using cosine similarity to compute similarities between items. However, for items with small amounts of data I'd like to bin them under a general "average" category (in the general not mathematical sense). To accomplish this I'm currently trying to create a synthetic observation to represent that middle of the road point.

So for example if these were my observations (rows are observations, cols are features):

[[0, 0, 0, 1, 1, 1, 0, 1, 0],
 [1, 0, 1, 0, 0, 0, 1, 0, 0],
 [1, 1, 1, 1, 0, 1, 0, 1, 1],
 [0, 0, 1, 0, 0, 1, 0, 1, 0]]

A strategy where I'd simply take the actual average of all features across observations would generate a synthetic datapoint such as follows, which I'd then append to the matrix before doing the similarity calculation.

[ 0.5 ,  0.25,  0.75,  0.5 ,  0.25,  0.75,  0.25,  0.75,  0.25]

While this might work well with certain similarity metrics (e.g. L1 distance) I'm sure there are much better ways for cosine similarity. Though, at the moment, I'm having trouble reasoning my way through angles between lines in high dimensional space.

Any ideas?


texts:
I am writing a code to classify images from two classes, dogs and cats. I wrote the below code, but always all the dogs images are classified as cats as shown in the confusion matrix.

Am I missing something in the code?

Note: I tried epochs up to 70, but got same results.

from keras.models import Sequential
from keras.layers import Conv2D,Activation,MaxPooling2D,Dense,Flatten,Dropout
import numpy as np
from keras.preprocessing.image import ImageDataGenerator
from IPython.display import display
import matplotlib.pyplot as plt
from PIL import Image
from sklearn.metrics import classification_report, confusion_matrix

classifier = Sequential()
classifier.add(Conv2D(32,(3,3),input_shape=(64,64,3)))
classifier.add(Activation('relu'))
classifier.add(MaxPooling2D(pool_size =(2,2)))
classifier.add(Conv2D(32,(3,3)))
classifier.add(Activation('relu'))
classifier.add(MaxPooling2D(pool_size =(2,2)))
classifier.add(Conv2D(64,(3,3)))
classifier.add(Activation('relu'))
classifier.add(MaxPooling2D(pool_size =(2,2)))
classifier.add(Flatten())
classifier.add(Dense(64))
classifier.add(Activation('relu'))
classifier.add(Dropout(0.5))
classifier.add(Dense(1))
classifier.add(Activation('sigmoid'))
classifier.summary()
classifier.compile(optimizer ='rmsprop',
                   loss ='binary_crossentropy',
                   metrics =['accuracy'])
train_datagen = ImageDataGenerator(rescale =1./255,
                                   shear_range =0.2,
                                   zoom_range = 0.2,
                                   horizontal_flip =True)
test_datagen = ImageDataGenerator(rescale = 1./255)

batchsize=32
training_set = train_datagen.flow_from_directory('/home/osboxes/Downloads/Downloads/dogs-vs-cats/train/',
                                                target_size=(64,64),
                                                batch_size= batchsize,
                                                class_mode='binary')

test_set = test_datagen.flow_from_directory('/home/osboxes/Downloads/Downloads/dogs-vs-cats/test/',
                                           target_size = (64,64),
                                           batch_size = batchsize,
                       shuffle=False,
                                           class_mode ='binary')
history=classifier.fit_generator(training_set,
                        steps_per_epoch =9000 // batchsize,
                        epochs = 30,
                        validation_data =test_set,
                        validation_steps = 4500 // batchsize)

Y_pred = classifier.predict_generator(test_set, steps=4500 // batchsize+1)
y_pred = np.argmax(Y_pred, axis=1)
print('Confusion Matrix')
print(confusion_matrix(test_set.classes, y_pred))
print('Classification Report')
target_names = test_set.classes
class_labels = list(test_set.class_indices.keys())
report = classification_report(target_names, y_pred, target_names=class_labels)
print(report)

# summarize history for accuracy
plt.plot(history.history['accuracy'])
plt.plot(history.history['val_accuracy'])
plt.title('model accuracy')
plt.ylabel('accuracy')
plt.xlabel('epoch')
plt.legend(['train', 'test'], loc='upper left')
plt.show()
# summarize history for loss
plt.plot(history.history['loss'])
plt.plot(history.history['val_loss'])
plt.title('model loss')
plt.ylabel('loss')
plt.xlabel('epoch')
plt.legend(['train', 'test'], loc='upper left')
plt.show()


texts:
I am new to jupyter notebook and i can't find my saved .ipynb code after logging out of jupyter notebook accidentally.
Thanks in anticipation


texts: score or cost function for AgglomerativeClustering

texts:
I am learning AgglomerativeClustering using sklearn. It is fairly easy to use for example:

  # create clusters
    hc = AgglomerativeClustering(n_clusters=10, affinity = 'euclidean', linkage = 'ward')
    # save clusters for chart
    y_hc = hc.fit_predict(points)

In my case,n_clusters is dynamic within a range, say from 9-15. I would like to run some cost function or score function, so I can plot a chart and then pick one there.

However, seems AgglomerativeClustering doesn't have a score function, unlike Kmeans, which I can either use inertia_ or score to plot.

So is there a way I can plot a chart and justify my n_clusters choice?


texts: Stratified K Fold Cross Validation in Orange: python script

texts:
I am using Orange to predict customer churn and compare different learners based on accuracy, F1, etc.
As my problem is unbalanced (10% churn - 90% not churn), I want to oversample. However, when using orange, this is not possible to do the oversampling within the cross-validation (test & score block).
Therefore, I want to, based on my input data, generate first 10 folds (stratified - where the distribution 10 % churn / 90 % not churn) is preserved. Then, oversample within each fold to get 50 - 50 distribution. Then, add for each instance the fold number as a feature. Lastly, within the test & score block, do cross validation by feature, namely the fold number. I think I have to implement this myself by using a Python script. Is there anyone that could help me doing this?
Thank you!
Emma


texts: Remove all characters following a certain character in a column of a dataset

texts:
I have a data set like the following, and the first column contains the groupings. However, some are labelled slightly differently. I need to remove all characters following the punctuation used (bracket, semicolon, comma).

groups <- c("Group1", "Group1", "Group1;Group1", "Group1(subset)", "Group1,ex" )

I would like this to present all of these just as Group1 (so they would all appear the same as the first two) - so to remove all characters in the string following the punctuation. I then need to repeat this for 1000s of groups, all in that same column of my dataset).

I know gsub is an option, but I'm not clear on how to use it to remove all of a string following a number of different characters, or on how to use it on just one column of a very large dataset.