texts: In a directed graph, how to measure whether a node is more "upstream" or "downstream"?

texts:
Let's say that I have a directed graph, and for each node I want to have some measure of whether it is more of an "upstream" node (that lies at the beginning of many paths), or a "downstream" node (to which many paths converge).

Say, in a sequence A->B->C I want A to have the highest value, and C the lowest value, while B would be "in between". In a perfect cycle A->B->C->A I want all nodes to have the same "in between" value. The metrics should tolerate a mix of cycles and acyclic elements.

Ideally, it would also give more extreme values to nodes from longer acyclic paths: higher values to nodes with larger estuaries (the 'mother of all' node), and lower values to nodes with larger watersheds (the 'all roads lead to Rome' node that integrates all flows).

And also ideally it should generalize to weighted graphs. And, ideally, in a way that does not depend on the scaling factor for weights (so if you multiply all weights in the graph by 2, the metrics for each node shouldn't change, as the topology clearly didn't change with this scaling).

For a non-weighted graph I can come with a metrics like that: find all predecessors of a node, find all children of a node, then divide the number of children to the sum of children and ancestors; this value would be 1 for any point of origin; would give 0 for dead-ends, and would give 0.5 for cycles. So it's mostly OK. What I don't like about this metrics is that 1) it does not care about the length of the path, 2) it will be computationally slow, 3) I don't know how to generalize it to weighted graphs in a scale-invariant way.

So I was wondering whether there's a known metrics with approximately these properties that was described and studied before. It feels like a logical thing to calculate that many people would use when analyzing social networks, for example; so it feels like it should have a name and published algorithms. Thank you!

Edit: I think it's fair to say that the pagerank metrics has many of the properties I described (with the values reversed): sinks are high, sinks with larger watersheds are higher, nodes of origin are low, cycles tend to have "in-between" values, and the algorithm clearly supports weighted graphs. The part it does not care about is whether a node of origin has a large estuary or no estuary at all. Now I'm wondering whether I actually need 2 metrics: one page-rank, for watersheds, and a different one for estuaries. Like a weighted share of nodes visited by random walks initiated in the node of interest, or something like that. Or are there simpler metrics?


texts:
TL:DR - Do you know of good automated tools to explore a dataset?

Long version: I have a few different datasets to work with from most various areas of business. I wonder if there are good software/scrips that could automatically round up the following answers for me:
Things I would be looking for:

datatype of a column?
dimension or measure?
histogram or count of values in a column
correlation between measures
unique key auto-find
import into a strongly typed data structure

Obviously I'm looking for something more than basic summary(cars). Something that goes through each column and spits out detailed analysis report


texts:
I am fairly new to learning the XGBoost algorithm and had a question about how the algorithm knows which feature to break the tree on first.
Here is my understanding (and please correct me if I'm wrong):
Each tree starts as a single leaf and all of the residuals go to that leaf. Then we calculate the similarity score and try to split amongst some feature characteristics. For example, say we only had one feature - height. We could split the residual if its respective x-value was say height > $180cm$ or height < $180cm$. And then proceed to calculate the similarity scores and the gain for the new leaves and continue down our tree.
My question is now say we had two or more features such as height, age, weight, education, etc. How does the algorithm know where to break the initial residuals? Does it compute it over all possible features and finds the best gain? I can imagine for large datasets to go through all features, then break it at some threshold and compare must take a long time.
Or does it start at some random feature and work its way down kind of like a random forest? I know there is a parameter within the algorithm that allows you to set the percentage of features per tree so adding on to my question once we specify this percentage how does it pick the features and more importantly once those features are picked how does it know which one to pick to break the initial leaf of residuals.
Edit: I tried reading the paper where the algorithm was first published but found it a bit too difficult to conceptualize and understand.


texts: backpropagation between fully connected layer and convolution layer?

texts:
This is a simple example of a network consisting of two convolutional layers and one fully connected layer.

target = 10x1
input  = 15x15
con_1  =  5x5 // filter size
con_2  =  3x3 // filter size
fc     =  10x81
// forward pass
out_1 = convolution(input,con_1) = 11x11
out_2 = convolution(out_1,con_2  = 9x9
out_3 = matrix multiplication(fc,out_2)
// backward pass
fc_err = err from fully connected layer
con_2_err = convolution(filter con_2,fc_err)// result 10x10
gradient = convolution(out_1,con_2_err) // result 2x2

After I find the error at the fc level, I have to pass it at conv_2 convolution level.
My concern is that the error is 10x1 in size and the con_2 filter is 3x3 in size.
So it should be?
If so, then I need to bring the size of the filter 3x3 to 10x10, and then the result of the convolution will be a 10x10 matrix.
The gradient for the layer con_2 will be the result of the convolution of out_1 and con_2_err, which will give a 2x2 matrix.
To update the weight of the con_2 filter, I need a 3x3 gradient.
What am I doing wrong ? Sorry for the large text and my English, I just want to understand why the size of the gradient does not match the size of the filter.


texts: Which data science model is best for explainability for prediction problems?

texts: Imagine you have to create a model to explain to stakeholders e.g. to predict price, weight, sales etc.. Which regression models offer the best in terms of explainability and interprability? ... Which model is best for this?


texts: How to compare two dataframes and put the counted unique values in a first dataframe's column?

texts: I have two different (geo)dataframes, one has 690 and the other has 1826 rows. The first one is a grouped based on the nearness (spatial near) of the second dataframe. Thus, they bound with FID_1 and NEAR_FID columns. Every FID_1 in the first dataframe corresponds to at least 2 NEAR_FIDs in the second dataframe. My challenge is to count the number of NEAR_FIDs in the second dataframe and put as a row in the first one. I am using python 3.5+. Thanks for any suggestion.


texts: Newbie: What is the difference between hypothesis class and models?

texts:
I am new to machine learning and I am confused with the terminology. Thus far, I used to view a hypothesis class as different instance of hypothesis function... Example: If we are talking about linear classification then different lines characterized by different weights would together form the hypothesis class.

Is my understanding correct or can a hypothesis class represent anything which could approximate the target function?  For instance, can a linear or quadratic function that approximates the target function together form a single hypothesis class or both are from different hypothesis classes?


texts: Dimensionality reduction based on value of a variable

texts:
I have a dataset including 100k high dimensional data (e.g. houses in LA) (dim=100, e.g. house parameters like area, distance to downtown, etc.). Below is the 2-component PCA representation of the data, colored by target value (e.g. housing price).

As you can see, price function has a stochastic behavior and changes rapidly everywhere.

Is there a dimensionality reduction method that takes the target value into account and as a result, it aggregates low priced homes on one side and we can see a gradient of housing prices in the map?
Can manifold learning help in anyway with this task?

I tried Backward feature extraction based on ridge regression to reduce the dimensionality based on their effect on regression results on target value, and then applied PCA and normalized it, but the output was not as desired. Here is the output:

Thanks


texts: Split large documents into many smaller ones?

texts:
I want to build a basic language detector for English, French and German.

I went to wikipedia and I downloaded the page of 'Technology' in all these languages.

In all these cases, we are talking for about 10000 words.

So basically I have 3 documents of 10000 words each for each of the 3 languages above.

My question is the following:

Should I split these documents in smaller documents e.g. of 100 words and create in this way more labeled observations in my dataset or should I leave them like this for training my classifier (e.g. with a TF-IDF model)?


texts:
I've read many explanation on how do to policy iteration, but I can't find an example, so I'm stuck right now trying to figure out to Policy Iteration.

The numbers next to each state show the reward received for arriving in that state. For example, if the agent started in $S_0$ and took the $Blue$ action, and ended up in state $S_1$, the immediate reward would be $-5$.

The discount value is 0.1, and the initial policy is $\pi(S_0)=Blue$ and $\pi(S_1)=Red$

$S_2 $ state is the terminal state - game over. The two possible actions are Blue and Red as can be seen on the image.

I just need something to help me start, because no explanation really made me understand how do I start the policy iteration till convergence.


texts: Longitudinal study - conversion long to wide - implications

texts:
I'm doing a longitudinal study and trying to find out patterns before a defined event happens. For instance, we have data:

ID Date Amount Feature Event_ind
1  2010 5      M       0
1  2011 4      M       0
1  2012 3      M       1

The objective is to 'predict' the time when there would be an event happening, given historical data.
Currently, we are exploring whether we can convert this long table to a wide table. The table would look like:

ID Date1 Amount1 Date2 Amount2 Feature
1  2010  5       2011  4       M

My question is, what are the implications of this conversion? (or any good blog/academic paper explaining this)

One of the implications that I can think of is, in a wide table each repeated measurements have its own distribution; and this conversion doesn't make sense if it's measured in different time or at different frequency.


texts: How to visualize ner dataset tagged using BILOU?

texts:
I have a dataset for ner which is tagged using BILOU tagging method and example of same is below
Minjun  B-Person
is  O
from    O
South   B-Location
Korea   I-Location
.   O

i wish to visualize how model learn if i use lstm with crfD
for example how this model is learning context of current sentence


texts: R Rpart taking too much time on data set

texts:
I am trying to build a CART model using  rpart on a data set with around 7k rows and 456 columns

cmodel2=rpart(DV ~ .,data=teltrain2,method="class")

This has never returned a result yet. I am using a 16GB ram machine with R 3.2.3. Every time on running this, R execution goes on and on, I have to kill the R GUI process to finally stop it.

Description of the columns

id --> unique id (which I have removed from the training set)
Location --> number
col3 to col400 --> these have been hot encoded by me and include just 0 or 1 as values
col400-col456 --> these are features which have numeric values like 23,144 etc, all less than 1000
DV --> dependent variable is categorical with more than 2 values.

I have tried running rpart with a smaller set of 1000 rows and 456 cols, still no luck. I don't want to do any kind of feature elimination on the first run as as of now I have no idea whether they affect the dependent variable or not.

I have tried mentioning minbucket=25,20,15,10 with no luck either.

My objective is to get the resulting model of rpart on the entire data set. What are my options?


texts: decision -tree regression to avoid multicollinearity for regression model?

texts:
I read in comments a recommendation for decision tree´s instead of linear models like neural network, when the dataset has many correlated features. Because to avoid multicollinearity.
A similar question is already placed, but not really answered.
https://stats.stackexchange.com/questions/137573/do-classification-trees-need-to-consider-the-correlation-between-attributes
or here
In supervised learning, why is it bad to have correlated features?
https://www.quora.com/Is-multicollinearity-a-problem-in-decision-trees#:~:text=Decision%20trees%20follow%20the%20non%20parametric%20approach.&text=Though%20single%20tree%20leads%20to,robust%20to%20the%20multi%20collinearity%20.
My problem:
I have a dataset of about 30 columns. 10 columns have a high correlation with the target/dependend variable. Data are numerical. I would like to do a prediction (regression model) include all variables if possible?
One big problem is to avoid multicollinearity.

Is there a decision tree regression model good when 10 features are high correlated? (if I follow the answers of the links, but there is no really good explanation for that).
Is there a scientific or math explanation or recommendation (to use decision tree regression)?


texts: Understanding computations of Perceptron and Multi-Layer Perceptrons on Geometric level

texts:
I am currently watching amazing Deep Learning lecture series from Carnegie Melllon University, but I am having little bit of trouble understanding how Perceptrons and MLP are making their decisions on a geometrical level.
I would really like to understand how to interpret Neural Networks on geometric level, but sadly I am not able to understand how computations of a single Perceptron relate to simple Boolean functions such as OR, AND, or NOT, which are all shown on picture below (e.g. what would be the required value of weights and input in order to model specific decision boundary).

Hopefully, if I were to understand how these computations relate to geometric view shown on picture above, I would be able to understand how MLPs model more complicated decision boundaries, such as the one shown on picture below.

Any help would be appreciated (concrete answer, reading resources, anything at all!). Thanks in advance!


texts: When to use missing data imputation in the data analysis problem?

texts:
I want to run statistical analysis of a dataset and build a logistic regression model and multinominal linear model by R according to the research question. But I was wondering which step should I use the missing value imputation to complete the dataset. I have finished the univariate analysis for each variable in the raw dataset, and I found there are three continuous variables and two categorical variables with lots of missing data. I want to use the missing data imputation to complete the dataset after processing with the bivariate analysis and graph exploration of every variable. But I am not sure if that's a correct order to do?

Should I use missing value imputation to complete the dataset before bivariate association analysis or should I do it after that?

In addition, if I want to examine the distribution of the outcome variables to find proper transformation, should I do it after impute the missing data as well?

Thank you!


texts:
I want a direction for ML technique to predict the next time you will be online in chat app

My table contains; user id, timestamp & status

status is a categorical variable (online, composing, offline), I'm saving a row for each status change which means user online, maybe writes something to me (composing) and afterwards went offline, so I can extract the time online (currently I don't see a reason for this feature)

I want to export the model to PMML

What is the best approach if I want to a service to get the next timestamp that user will be online ?


texts: Is there a way (like planning algorithms) to draw a successive polyline to fill a specific shape (like triangle)?

texts:
there is a specific shape (like triangle) area, i am trying to develop an program to draw a successive polyline inside the triangle to fill the triangle, one line 3 layers

this post demonstrates some algorithms to solve this category of problems.

none of them is suitable for this task, any ideas?


texts: Should use sklearn or tensorflow for neural networks?

texts: I have just started learning Neural Networks for deep learning from cs231. I am trying to implement Neural Network in Python. I am looking at using Tensorflow or scikit-learn.  What are some pros and cons of these libraries for this application?


texts:
This is from a referee report in a conference to which I submitted my paper - I don't quite get it and I'm not sure what I need to do about it.

I use Euclidean loss and Softmax cross-entropy (pixelwise) loss in my model. The referee pointed out that the losses are not in the same range and they are not weighted.

I've never seen that done in other models (e.g. Faster R-CNN, Mask R-CNN). Any suggestions how to address this issue?


texts:
I am having problems with very big texts.

after preprocessing each of my docs represented as a matrix of sentences, where each sentence represented as encoded words (each word have a unique vocab value that represent it).

I have two types of docs, some are very big and some are very small.
so in the end there is two types of matrices :
let x1 be matrices of MxN and x2 be matrices of m x n , I want to take all the x1 matrices and make to then dim reduction from MN to mn is it possible? and if so how?

I am working with python and ***I wanted to do it with svd, the problem is that I only have managed to do reduction of N to n , but could not reduce the rows.

any help will be appreciated.


texts:
I'm looking for references to a standard(-ish) task/dataset in NLP that is close(-ish) to the following: we have a document with a list of references (sorry), for example, a scientific paper. For every reference, we have a sentence within the document and the corresponding text of another paper. The task is to find sentences/regions within the corresponding text, that "support" the sentence in the original text, given some metric. Basically, what anyone would do given cumbersome unexplained statement within some mind-boggling paper.

Apart from the main question, what are the possible/good/standard metrics one can use? Are there remarkable implementations/users of some in-house solutions/any other useful remarks?


texts: Avoid recycling when merging with R Data Table

texts:
I got tableA, which contains a numeric variable, CdTaller (e.g. 1948675).

I also got tableB, which contains two variables: Id (e.g. 2513978) and Name (e.g. "JOSE MANUEL PEREZ")

I want to add a new column to tableA, Name_Taller, which contains a name for those CdTaller which correspond to the Id variable of tableB, so I do the following:

tableA[, Name_Taller := tableB[Id==tableA\$CdTaller]$Name]

However, for those CdTaller of tableA that are not in tableB, R recycles the new variable Name_Taller:

Warning message:
In \`[.data.table\`(tableA, , \`:=\`(Name, tableB[Id ==  :
  Supplied 867324 items to be assigned to 947871 items of column 'Name'
  (recycled leaving remainder of 80547 items).

So for the tableA$CdTaller == '1320402', which can't be found in tableB, R still adds a Name_Taller which actually corresponds to CdTaller == '4430219', because of this recycling.

The question is: how can I avoid this recycling, so that R only adds a NA or empty string to those rows containing a CdTaller not found in tableB? Is there any parameter in data table I can use for this purpose?


texts: RCNN - recurrent convolutional neural networks - what is a time step?

texts:
I am currently studying this paper for implementing it into a speech recognition application, but seem to have some problems understanding the concepts of time step.  What is a time step?

I haven't worked with RNN before but would guess that in a continuous stream of data time step would be the data present at time t.

but how do one even implement/train something like that in an iteration based system?...

I mean something like keras, one is not able to involve output received from t-1 to t... or if so, how?, I seem to be unfamiliar with this...

In another paper i found this

  To understand the essen ce ofthe RCL, it is useful to c1arifY the
  concept oftime step in RCL. It is not identical to the time associated
  with the sequential data, and instead it refers to an iteration during
  processing the data.

from RECURRENT CONVOLUTIONAL NEURAL NETWORK FOR SPEECH PROCESSING

which then further conforms my theory of it being being about the data that was involved/processed as iteration t-1.


texts: Using Discretization from Training Set on Test Set in R

texts:
I am currently discretizing my training set in R with discretize from the bnlearn package.

library(bnlearn)
discretize(train, method = "quantile", breaks = 2)

The lower bound of one interval and the upper bound of the other interval are the minimum and the maximum of the respective column.

Executing the same command on the test set

discretize(test, method = "quantile", breaks = 2)

will result in a different discretization, as the minimum and the maximum will likely be different on the test set. Is it possible in R (with another library or command) to transfer a discretization from a training set to a test set?

Apparantly this is easy to do in Weka and Orange, however, I would prefer to do this in R (not using RWeka).


texts: Computing confidence interval of average output from random forest in scikit learn

texts:
I am computing confidence intervals for random forests using the package available here: http://contrib.scikit-learn.org/forest-confidence-interval/auto_examples/plot_mpg.html#sphx-glr-auto-examples-plot-mpg-py

I want to plot the average MPG and associated confidence interval. I can compute the average MPG but how do I compute the associated confidence interval?


texts: How can I intuitively calculate the accuracy of my financial prediction model?

texts:
I've built a SARIMAX model based on my personal spendings record as a college thesis and have reached a point where I'm pretty content with how it turned out and am getting ready to start writing the documentation.
However, besides technical details, I have to provide an easy to understand view on the system's performance to people unfamiliar with machine learning.
At the moment I've implemented an error function like this: MAE_i = abs(predicted_i - observed_i) / observed_i and then I calculate the sum of parts and average them out MAE = sum(MAE_i) / len(observations). This gives me an error of about 7%.
Is this acceptable or can I devise an easier to understand view on the accuracy of my model?
def computeError(pred, originalData):
    slice = originalData.loc['2019-08-25 00:00:00':]
    meanError = 0
    for pair in zip(slice.values, pred):
        meanError += abs(pair[0] - pair[1]) / pair[1]

    print(f'Mean Absolute Error is {meanError/len(pred)*100}%')
```


texts:
I have a data set of districts, farmland area and fertilizer subsidies issued for those areas. I.e, using made up numbers,

district | area | subsidy | subsidy per area (computed)
abc      |   20 |   500   |         25
cde      |   30 |   750   |         25
fgh      | 0.02 |    15   |        750    <--- looks off

I'm trying to visualise the subsidy per area but in districts that have very small amounts of farming the subsidy per area seems abnormal. The nationwide average is pretty much around 25. So, I can safely say that the subsidy amount is directly related to the area being subsidised, which is to be expected as fertiliser usage is dependent on the area being farmed. My theory is that the exception on small areas is due to there being a minimum subsidy amount irrespective of the land area.

Are there any techniques to deal with the above scenario when visualising data?


texts: seaborn heatmap - x axis - repeated values

texts:
I'm in trouble creating a heatmap using a CSV file.
csv data is in a format like below

here is a code
years = np.array(datadf.PublicationYear)

sns.set(font_scale=2)
wordlist = ['greenhouse_gas', 'pollution', 'resilience', 'urban','city', 'environmental_impacts', 'climate_change',
            'adaptation','mitigation','carbon', 'ghg_emissions','sustainable','sustainability','lca']

word_tuples = [('urban','city','urban'), ('greenhouse_gas','ghg_emissions','greenhouse_gas'),('sustainable','sustainability','sustainability')]

use_wordlist = True
word_number = 30
freqdata = []
agg_keys = []

for i in np.arange(len(abstrct)):
    ngram_model = Word2Vec(ngram[[abstrct[i]]], size=100, min_count=1)
    ngram_model_counter = Counter()
    for key in ngram_model.wv.vocab.keys():

        if key not in stoplist:
            if use_wordlist:
                if key in wordlist:
                    if len(key.split("_")) > N:
                        ngram_model_counter[key] += ngram_model.wv.vocab[key].count
            else:
                if len(key.split("_")) > N:
                        ngram_model_counter[key] += ngram_model.wv.vocab[key].count

    freqdf = pd.DataFrame(ngram_model_counter.most_common(word_number))
    if len(freqdf.index) == 0:
        freqdf[0] = wordlist
        freqdf[1] = 0

    for w in word_tuples:
        if w[0] in wordlist and w[1] in wordlist:
            f = 0
            drops_w = []

            for j in np.arange(len(freqdf.index)):
                if freqdf.iloc[j][0] == w[0] or freqdf.iloc[j][0] == w[1]:

                    f += freqdf.iloc[j][1]
                    drops_w.append(j)

            freqdf = freqdf.drop(drops_w, axis = 0)
            append_data = pd.DataFrame({0:[w[2]],1:[f]})
            freqdf = freqdf.append(append_data,ignore_index=True)
            freqdf = freqdf.reset_index(drop=True)

    #Normalizing the frequency by the total number of non-stopword tokens
    freqdf['prob'] = freqdf[1]/(len(abstrct[i]))

    agg_keys += np.array(freqdf[0]).tolist()
    freqdata.append(freqdf)

unqkeys = np.unique(np.array(agg_keys))

matrix = np.zeros([unqkeys.size,len(freqdata)])

for i in np.arange(years.size):

    for j in np.arange(unqkeys.size):

        for k in np.arange(len(freqdata[i].index)):

            if freqdata[i].iloc[k][0] == unqkeys[j]:
                matrix[j,i] = freqdata[i].iloc[k]['prob']

fig, ax = plt.subplots(figsize = (40, 26))

ax = sns.heatmap(matrix, annot = False,linewidths = .9,cmap = 'Blues' ,cbar_kws={'label': 'Scaled Frequency'})
ax.figure.axes[-1].yaxis.label.set_size(35)

ax.grid(False)

ax.set_yticks(np.arange(len(unqkeys))+0.5) #Adding 0.5 offset
ax.set_xticks(np.arange(len(years))+0.5)
ax.set_yticklabels(unqkeys,rotation= 0, fontsize = 34.0)
ax.set_xticklabels(years,rotation='vertical', fontsize = 35.0)
ax.set_xlabel('Publication Year', fontsize = 40.0, labelpad = 40)
ax.set_ylabel('Keywords',fontsize = 40.0, labelpad = 5)

plt.show()
image_path =  os.path.join(base_dir, 'plots/') + data_dir + '-heatmap.jpg'
fig.savefig(image_path)

It generates a graph below. How could it make x axis like [2016,2017,2018,2019,2020,2021]


texts:
I'm a beginner with ANN and DL in general. I have a regression task with a target of 2-dimensions, my dataset only have 46 samples (small dataset I think).
I tried the code below that does a regression with only one output which works normally.

When I change to a two dimensional regression, my loss function becomes equal to NaN.
I tried to change the optimizer and fix the dropout rate, but nothing changed, any solution?

import numpy
import pandas
from keras.models import Sequential
from keras.layers import Dense
from keras.layers import Dropout
from keras.wrappers.scikit_learn import KerasRegressor
from sklearn.model_selection import cross_val_score
from sklearn.model_selection import KFold
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline

#load data
dataframe = pandas.read_excel("data1.xlsx")
dataframe.isnull().any()

dataset = dataframe.values

X = dataset[:, 0:5]
Y = dataset[:,5:7]

def baseline_model():
    #create model
    model = Sequential()
    model.add(Dense(10, input_dim=5, kernel_initializer='normal', activation
='relu'))
    model.add(Dropout(0.5))
    model.add(Dense(2, kernel_initializer='normal'))

    #compile model
    #model.compile(loss='sparse_categorical_crossentropy', optimizer='adam')
    model.compile(loss='mean_squared_error', optimizer='adam')
    return model

#fix random seed for reproducibility
seed = 7
numpy.random.seed(seed)

estimators=[]
estimators.append(('standardize', StandardScaler()))
estimators.append(('mlp', KerasRegressor(build_fn=baseline_model, epochs=50, batch_size=32, verbose=0)))
pipeline = Pipeline(estimators)

kfold = KFold(n_splits=10, random_state = seed)
results = cross_val_score(pipeline, X, Y, cv=kfold)

print ("Result: %.2f (%.2f) MSE" %(results.mean(), results.std()))


texts: Conceptual question on CNN and any multi layer neural network (Part 2)

texts:
I have read a number of tutorials and online lectures (https://ujjwalkarn.me/2016/08/11/intuitive-explanation-convnets/) but none of them mention the rationale for selecting a particular design. How do we decide on the following design aspects?

1) Is there a rule of thumb for deciding on the number of layers?  Or is it purely on the basis of trial and error?

2) Can somebody please explain the intuition and the rationale for designing a CNN architecture for this example -- considering a binary classification problem. For an input RGB image of size 500*500*3, how would you design the architecture -- how many layers, number of filters, size of the filter, how much is the stride, etc.


texts: What is the difference between explainable and interpretable machine learning?

texts:
O’Rourke says that explainable ML uses a black box model and explains it afterwards, whereas interpretable ML uses models that are no black boxes.

Christoph Molnar says interpretable ML refers to the degree to which a human can understand the cause of a decision (of a model). He then uses interpretable ML and explainable ML interchangably.

Wikipedia says on the topic "Explainable artificial intelligence" that it refers to AI methods and techniques such that the results of the solution can be understood by human experts. It contrasts with the concept of the black box in machine learning where even their designers cannot explain why the AI arrived at a specific decision. The technical challenge of explaining AI decisions is sometimes known as the interpretability problem.

Doshi-Velez and Kim say that that interpretable machine learning systems provide explanation for their outputs.

Obviously, there are a lot of definitions but they do not totally agree. Ultimatively, what should be explained: The results of the model, the model itself or how the model makes decissions? And what is the difference between interpret and explain?


texts: Multiple Features with the same categorical value

texts:
In working with a telecommunications data set with multiple categorical variables which all depend on Internet Service, a separate categorical variable, I ran into the problem where 'No Internet Service' is dummy encoded several times in my training data. See the below pie charts for an illustration of the problem.
I was wondering what can be done to avoid this repetition of 'No Internet Service' as a feature value across several features, as it increases the dimensionality of my training data significantly and makes all of my features very highly interdependent.
Feature selection after encoding helps, but often leaves just the dummy variable 'No Internet Service' when removing the categories which don't contribute to the label.


texts: Predicting probability for each tag given already chosen tags

texts: I have a set of tags (~10'000, will be extended over time) presented to a user. After he has selected 3 or more tags, I want to predict for each remaining tag what the chances are that the user will select this tag as well. I strictly need the prediction for all remaining tags. The prediction should be fast but accuracy isn't that important. The training data would be cases where users already have selected a subset of tags. What would be a good approach here?


texts: Storing Large dataset for processing and analysis of data

texts: I am new to data engineering and wanted to know , what is the best way to store more than 3000 GB of data for further processing and analysis ? I am specifically looking for open source resources . I have explored many data formats for storage . The dataset that I want to store is a heart rate pulse data generated by a sensor.


texts: Is machine learning the right tool for this job?

texts:
I would like to create a troubleshooting wizard.

The user will go through the wizard and choose different options, what options they choose will determine what is displayed next in the wizard.

Eventually the user will solve their problem (or not), at the end they will choose 'yes' or 'no' if there problem was solved.

The wizard would learn what most common resolutions to a problem was based on past input.

Before I go exploring machine learning further, does this sound like the right tool for the job?


texts: Significance shifts when cases are excluded

texts:
I have a county level dataset with a dichotomous variable which represents whether or not any charter schools exist in a county.  I am performing a nested logistic regression. That variable is the response and the explanatory variables are social factors such as education level, language spoken at home, household type, etc.  When I include all counties in the U.S. (Including ten states which have no charter authorization laws and thus every county in those states is represented as a zero in the response variable) my results show significant associations between the response and the explanatory variables.  When I exclude those ten states the significance disappears.

Any ideas as to why this might be happening?


texts: What is the difference between ensemble methods and hybrid methods, or is there none?

texts:
I have the feeling that these terms often are used as synonyms for one another, however they have the same goal, namely increasing prediction accuracy by combining different algorithms.

My question thus is, is there a difference between them? And if so is there some book/paper that explains the difference?


texts: Passing the output of a model as input to an other model , computer vision?

texts:
i want to try my hands at a project where i have to detect the brands + the models of sportswears worn by persons participating to a marathon , the project is quite complex , so my first approach would be to use an already pre-trained model that will draw the bounding box of the persons , then use the outputed bonding boxes as input to an other model that will draw me the bounding box for the shirt , shoes , shorts ...ect , and follow this procedure until i get to the brand and then the model of the sportswear.
theorically , it's super simple , the problem is that , i'm having a hard time figuring out how to implement it by code , passing the output of one model as an input to an other model , is there any good exemples on the internet can illustrate that approach , by code i mean ?

Any suggestion would be much appreciated , thank you.


texts:
I have a project where I'm required to compute (using some regression) how long a task will take. From the definition of the business problem it is clear that there is some temporal relationship in the data so I need to split into train/test by some cutoff date (rather than randomly sampling 70/30 split from the data).
The issue I'm running into is that no matter what date I split by, the distribution of my target variable is different in the testing & training set. Earlier targets in the set are a lot larger, and as a result the model fit on the training set tends to way over-predict on the test set.
Any advice on how to tackle something like this?


texts: What is the State-of-the-Art open source Voice Cloning tool right now?

texts: I would like to clone a voice as precisely as possible. Lately, impressive models have been released that only need about 10 s of voice input (cf. https://github.com/CorentinJ/Real-Time-Voice-Cloning), but I would like to go beyond that and clone a voice even more precisely (with subsequent text-to-speech using that voice). It doesn't matter if I have to provide minutes or hours of voice inputs.


texts: Generating data that look alike my original data

texts:
If I have a set of N data, each individual data has 4 features. I do not know a priori the relations that could exist (or don't exist) between the features.

Is it possible to generate new data from my initial set that will respect the implicit relations that could exist between the features without having to find explicitly these relations ?


texts: Real-life applications/examples of transfer learning approaches

texts:
I recently read a nice, informative paper titled 'A Survey on Transfer Learning'. It mentions 3 settings of transfer learning - inductive, transductive, and unsupervised. At the same time, it states there are 4 common approaches on "what knowledge to transfer" for each setting - instances, features, model parameters, or relations.
I looked up the descriptions of each category, but I do not understand exactly how they are utilised in the domains of Computer Vision and NLP. Could I get an example application of each of these categories to understand better how exactly they differ from each other? How does one determine which approach to use in which setting?


texts: When does it make sense to add numbers with different units?

texts:
Given two vectors containing numbers that have different natures / units, (example length in Meters and weight in Kilograms), does it make sense to calculate euclidean distance between these two vectors or cosine similarity?
The equations imply that you have to add $meters^2$ and $kilometers^2$ which supposedly does not make sense. Yet, I see this done many times indirectly, for instance, when calculating cosine similarity of documents based on tf-idf (vector that contains objects with dissimilar natures).


texts: How to randomly sample crops from plain image with points only if crop contains n points inside?

texts:
Lets assume I have an image which only has white background and black points, all same size. I need to randomly sample crops with a hardcoded size. The condition is that all the crops need to contain at least  n points inside. Also, ideally the crops can be rotated any number of degrees. In the following sample image I have manually drawn what it would be considered as valid crops (green) and non-valid crops (red). The condition in this example is that a valid crop must have at least 5 points inside.

Note that I have made the image with paint so the crops are not exactly same size (they should), an also there are only two rotations (there should be any type of rotation)

I am looking for an efficient algorithm for doing this and my idea is to implement it in Python. Any suggestions?

Thanks


texts:
I am using chi-squared to determine feature importance as I select features to train a supervised ML model. I create a contingency table for the feature/target, and feed this contingency table into the scipy.stats.chi2_contingency module. This module returns the chi-squared value and the p-value.

I have acheived reasonable results with boolean variables, but I am suspicious of the results for categorical variables with more than 2 categories.

Specifically, I am fairly sure that one continuous feature, age, is correlated with the target, to some level of significance. From plotting histograms and KDEs, I know that the probability distribution of the feature for (target = 0) is quite different from the probability distribution for (target = 1). However, when I bin the age feature into 2-7 bins, the chi-squared test yields a p-value of ~1e-39.

Is there anything that I am missing with regards to the chi-squared test and categorical variables? Does this test only work for monotonic relationships?


texts: Support vector regression and paremeters

texts: I am doing load forecasting using SVR(kernel='rbf').How can I understand which is the best value for parameters C, epsilon and gamma?Thanks.


texts: NLP: find the best preposition for connecting parts of a sentence

texts:
My task is to connect 2-3 parts of the sentence into one whole using a  preposition

the first part is some  kind of action. Ex. "take pictures"
the second part is an object  that can consist of  only one noun or a noun with  adjectives and additions dependent on it. Ex. "juicy cherry pie", "squirrel"
the third part is a place. Ex. "room", "London"

To solve this task I've already tried some options  such as generation using GPT-2 (or other networks like LSTM) and using grammar rules, but both of them didn't work as well as I want.
Using GPT-2 needs  a lot of resources and performance, which is very expensive on the server. Ordinary neural networks aren't good enough and often generate bad results. And it is also impossible to describe all cases using only grammar rules.
Are there any approaches how you can implement a solution to this problem without using transformers or grammatical rules, so that the model isn't too heavy and at the same time effective?
There may be several options for prepositions, the main thing is that they are used.
P.s. a good option was to use n-grams, but it is not clear how to work with it when the second part is long.
Examples:
[Pour, juice, boiler room]     ->  Pour the juice into the boiler room
[Go, theater]                  ->  Go to the theater
[Learn the history, ballet]    ->  Learn the history of ballet


texts: Why is the "dying ReLU" problem not present in most modern deep learning architectures?

texts:
The $ReLU(x) = max(0,x)$ function is an often used activation function in neural networks.
However it has been shown that it can suffer from the dying Relu problem (see
also What is the "dying ReLU" problem in neural networks?)
Given this problem with the ReLU function and the often seen suggestion to use a leaky ReLU instead, why is it that to this day ReLU remains the most used activation function in modern deep learning architectures? Is it simply a theoretical problem that does not often occur in practice? And if so, why does it not occur often in practice? Is it because as the width of a network becomes larger the probability of dead ReLUs becomes smaller (see Dying ReLU and Initialization: Theory and Numerical Examples
)?
We moved away from sigmoid and tanh activation functions due to the vanishing gradient problem and avoid RNN's due to exploding gradients but it seems like we haven't moved away from ReLUs and their dead gradient? I would like to get more insights in to why.


texts: Can someone please explain what this sample function is upto?

texts:
So there is a function in Dino_Name_Generator at Deeplearning.ai notebook

def sample(parameters, char_to_ix, seed):
    # Retrieve parameters and relevant shapes from "parameters" dictionary
    Waa, Wax, Wya, by, b = parameters['Waa'], parameters['Wax'],parameters['Wya'], parameters['by'], parameters['b']
    vocab_size = by.shape[0]
    n_a = Waa.shape[1]

    ### START CODE HERE ###
    # Step 1: Create the one-hot vector x for the first character (initializing the sequence generation). (≈1 line)
    x = np.zeros((vocab_size, 1))

    # Step 1': Initialize a_prev as zeros (≈1 line)
    a_prev = np.zeros((n_a, 1))

    # Create an empty list of indices, this is the list which will contain the list of indices of the characters to generate (≈1 line)
    indices = []

    # Idx is a flag to detect a newline character, we initialize it to -1
    idx = -1

    # Loop over time-steps t. At each time-step, sample a character from a probability distribution and append
    # its index to "indices". We'll stop if we reach 50 characters (which should be very unlikely with a well
    # trained model), which helps debugging and prevents entering an infinite loop.
    counter = 0
    newline_character = char_to_ix['\n']

    while (idx != newline_character and counter != 50):

        # Step 2: Forward propagate x using the equations (1), (2) and (3)
        a = np.tanh(np.dot(Wax, x) + np.dot(Waa, a_prev) + b)
        z = np.dot(Wya, a) + by
        y = softmax(z)

        # for grading purposes
        np.random.seed(counter+seed)

        # Step 3: Sample the index of a character within the vocabulary from the probability distribution y
        idx = np.random.choice(vocab_size, size=None, p = y.ravel())

        # Append the index to "indices"
        indices.append(idx)

        # Step 4: Overwrite the input character as the one corresponding to the sampled index.
        x = np.zeros((vocab_size, 1))
        x[[idx]] = 1

        # Update "a_prev" to be "a"
        a_prev = a

        # for grading purposes
        seed += 1
        counter +=1

    ### END CODE HERE ###

    if (counter == 50):
        indices.append(char_to_ix['\n'])

    return indices

Can someone please help and explain what benefit of returned indices over normal char_to_integer indices?

I want to understand the text processing in the link carried out before feeding into the network.


texts: How can I use a validation set to tune the hyperparameters of an XGBClassifier?

texts:
I'm currently building a ranking model using an XGBClassifier.
I have training, testing, and validation sets. I want to use the validation set to tune the hyperparameters of the XGBClassifier before using it to train a model. Is this the right idea?
Currently, I'm getting a 56% accuracy score with the default XGBClassifier (or 51% accuracy if I run PCA before training the model, which greatly reduces the time it takes to train). If I try tuning the hyperparameters before training, the results do not get any better at all. Here is what I'm doing:
First, I try tuning the hyperparameters with a RandomizedSearchCV. The validation features and class labels have been gathered before this code is executed.
params = {
    "learning_rate"   : [0.05, 0.10, 0.15, 0.20, 0.30],
    "max_depth"       : [3, 4, 5, 6, 8, 10, 12, 15],
    "min_child_weight": [1, 3, 5, 7],
    "gamma"           : [0.0, 0.1, 0.2, 0.3, 0.4],
    "colsample_bytree": [0.3, 0.4, 0.5, 0.7]
}
classifier = XGBClassifier()
random_search = RandomizedSearchCV(classifier, param_distributions=params, n_iter=5, n_jobs=-1, cv=5, verbose=3)
scaling = MinMaxScaler(feature_range=(-1, 1)).fit(validation_features)
validation_features = scaling.transform(validation_features)
random_search.fit(validation_features, validation_labels)
print(random_search.best_estimator_)

I then take the parameters shown by the previous print statement and plug them into my XGBClassifier, like so:
classifier = XGBClassifier(base_score=0.5, booster='gbtree', colsample_bylevel=1,
              colsample_bynode=1, colsample_bytree=0.7, gamma=0.4,
              learning_rate=0.15, max_delta_step=0, max_depth=5,
              min_child_weight=3, missing=None, n_estimators=100, n_jobs=1,
              nthread=None, objective='multi:softprob', random_state=0,
              reg_alpha=0, reg_lambda=1, scale_pos_weight=1, seed=None,
              silent=None, subsample=1, verbosity=1)

pca = decomposition.PCA(n_components=3)
pca.fit(train1_features)
train1_features = pca.transform(train1_features)
test1_features = pca.transform(test1_features)
scaling = MinMaxScaler(feature_range=(-1, 1)).fit(train1_features)
train1_features = scaling.transform(train1_features)
test1_features = scaling.transform(test1_features)
score = get_score(classifier, train1_features, test1_features, train1_labels, test1_labels)

Is there something glaringly obvious I'm doing wrong here? I'm new to machine-learning. I've mainly been following ideas I've seen online. Should tuning the hyperparameters always show better results, or are there cases where it does not help? Any advice is appreciated!


texts: Why is it good news that the expected gradient of the loss for a mini batch is equal to the gradient for the whole set?

texts:
I recently proved that

$\mathbb{E}_I[\nabla L_I(x,y,w)] = \nabla(L_I(x,y,w))$

Where $I$ represents a randomized batch of size $m$.

However, I cannot really understand why this is good news. I know that gradient is important for optimizing a model, especially with gradient descent. I also understand that in mini-batch algorithms we average over a whole batch instead of doing it for all the data, saving a lot of computations and reducing the variance.


texts: Modelling regressor of historic data on basic features test set

texts:
I am using a historical dataset of sales of items in shops and I need to predict the sales of the next month of the period.

I performed feature engineering, and now I have 10 features in the train set - median, avg of sales (prices and quantities).
I want to perform Linear Regression, The test set contains only 2 features - Shop_id and Item_id. How can I perform prediction from fitted regressor with 10 coefficients, and the test has only the 2?

I am not talking about doing PCA - the historic data has few historic sales  features and the test only has the basics

Dataset: Competitive data science predict future sales


texts: Which one is better method and why? Manually Handcrafted sound features vs spectrogram + convolution

texts:
I am working on classifying different sounds ( not speech or words exactly something like ambulance alarm, police alarm, cough sounds etc)
I read few paper which suggested to extract dsp features such as MFccs, skewness, kurtosis, log energy, entropy, zcr etc ( using total around 20 features ) from each segmented sound event.
I am currently using all these features and training xgboost, 3 layer DNN with ReLU, I am getting good accuracy.
I also read recent deep learning papers where they just used spectrogram images and feed it to convolution network which is capturing both temporal and spatial features. ( Using spectrogram one feature only )
I am looking for some explanation which one is better method for classifying sounds and why?
Any reference paper for comparison would be additional help.
Thank you!


texts: Is the k-center problem part of machine learning or computational geometry?

texts:
I am currently working especially with the k-center problem, which is e.g. used to determine optimal locations for k warehouses. This is done by defining k circles that cover a given set of points (here for example inhabitants), and at the same time, the radii of the circles should be minimal. The centers of these k circles would then be the optimal locations for the warehouses.

The question arises whether this method belongs to machine learning or computational geometry. Since it is a clustering method I would actually think that it belongs to machine learning, more specifically to unsupervised learning. On the other hand, there are many algorithms to solve this problem which are based on computational geometry.

How about your opinion?


texts: Visualization Tools that can show proper branch and merge in Graphs

texts:
I want help finding the visualization tool can draw similar architecture as given in the image below

Keras visualization produces something similar to this graph. But I'm working in Pytorch. I tried using Netron and Graphviz, both produce graphs that do not show the branching and merging properly.

This is the plot I rendered in Graphviz for another branching and merging architecture.

Here's the python code I used to render the plot

graph {
    graph [rankdir=RL]
    input [label=Input]
    conv1 [label="Conv 1"]
    conv2 [label="Conv 2"]
    conv3 [label="Conv 3"]
    conv4 [label="Conv 4"]
    conv5 [label="Conv 5"]
    conv6 [label="Conv 6"]
    conv7 [label="Conv 7"]
    conv8 [label="Conv 8"]
    conv9 [label="Conv 9"]
    batch1 [label="Batch Norm1"]
    batch2 [label="Batch Norm2"]
    batch3 [label="Batch Norm3"]
    caps1_a [label="Caps 1(a)"]
    caps1_b [label="Caps 1(b)"]
    caps2_a [label="Caps 2(a)"]
    caps2_b [label="Caps 2(b)"]
    caps3_a [label="Caps 3(a)"]
    caps3_b [label="Caps 3(b)"]
    sum1 [label=Sum]
    sum2 [label=Sum]
    sum3 [label=Sum]
    stack [label=Stack]
    sum4 [label=Sum]
    softmax [label=Softmax]
    input -- conv1 [constraint=true]
    conv1 -- conv2 [constraint=false]
    conv2 -- conv3 [constraint=false]
    conv3 -- conv4 [constraint=false]
    conv4 -- conv5 [constraint=false]
    conv5 -- conv6 [constraint=false]
    conv6 -- conv7 [constraint=false]
    conv7 -- conv8 [constraint=false]
    conv8 -- conv9 [constraint=false]
    conv3 -- batch1 [constraint=false]
    conv6 -- batch2 [constraint=false]
    conv9 -- batch3 [constraint=false]
    batch1 -- caps1_a [constraint=false]
    batch2 -- caps2_a [constraint=false]
    batch3 -- caps3_a [constraint=false]
    caps1_a -- caps1_b [constraint=false]
    caps2_a -- caps2_b [constraint=false]
    caps3_a -- caps3_b [constraint=false]
    caps1_b -- sum1 [constraint=false]
    caps2_b -- sum2 [constraint=false]
    caps3_b -- sum3 [constraint=false]
    sum1 -- stack [label=w1 constraint=false]
    sum2 -- stack [label=w2 constraint=false]
    sum3 -- stack [label=w3 constraint=false]
    stack -- sum4 [constraint=false]
    sum4 -- softmax [constraint=false]
    input -- conv2 [label=k1 constraint=false]
    conv2 -- conv4 [label=k2 constraint=false]
    conv4 -- conv6 [label=k3 constraint=false]
    conv6 -- conv8 [label=k4 constraint=false]
}


texts: read csv file directly from URL / How to Fix a 403 Forbidden Error

texts:
The csv file is downloadable. I can download the file and use read_csv, But I want to read the file via direct URL in jupyter, I used the following code, but I get the HTTP 403 Forbidden
 error

from io import StringIO

import pandas as pd
import requests
url="https://fineli.fi/fineli/en/elintarvikkeet/resultset.csv"
s=requests.get(url).text

c=pd.read_csv(StringIO(s))
c

how do I read the csv file via URL directly in python with a delimeter ";"


texts:
In Keras, the first object returned in the score list is loss.

score = model.evaluate(X_test, y_test,verbose=1)
print(score)

[1] [0.025217213829228164, 0.99487179487179489]

Is this the same thing as variance?


texts: Iteratively improving ML model on a small dataset

texts:
I have a spam classification model which I created using a very small dataset.I have exported it as shown
#saving model
import joblib

joblib.dump(spam_classifier, '../models/spam-detection-model.pkl')

Since I have limited computation power I want to retrain it such that it improves upon the new data as well. You can assume this problem as creating a model using very large dataset which is divided into chunks and then feeding a new chunk of data each day. All I know is this
model = joblib.load(./models/spam-detection-model.pkl')
# Code to train on new data such that it retains what is learned from previous chunk as well.
# Is something like this possible
improved_model = model.fit_and_improve_while_retaining_old_knowledge('new data')


texts: I would like to ask you why there are no Volcano-plot or MA-plot widgets in bioinformatics add-on already? As I can see it in the youtube video tutorial (two years ago, version 3.3). In the latest version, a functionality of making Volcano-plot disappeared. My question is why and or where?


texts: How can i test the performance of a model when the test data contains seen and unseen data

texts:
To test the performance of my model based on some selected features, i try to use unseen and seen data. However, when choosing the accuracy  based on all data, the model is almost overfitting since the size of seen data is larger than the other. When i take only the unseen data. The choosen features do not guarantee that the seen data are properly predicted. Is there any way to surpass this issue.

Thanks


texts:
I have a Eurobarometer dataset and want to eventually create a logistic regression model and a linear probability model using a set of dates as the dependent variable.
However, the dates in the dataset are displayed as long descriptors such as "Sunday 21st May 2005" in a column labelled v584 .

How can I create a dummy variable to say 0 = before Sunday 29th May 2005, 1 = after Sunday 29th May 2005 from the dates listed?

And how can I include only specific date (E.g., only including the dates a week before "Sunday 29th May 2005" in 0 and only the dates a week after in 1)?


texts: What is the difference between sklearn's transform() and predict() methods?

texts:
What I have understood thus far is that transform() gives you values that are qualitatively similar to X (your features) and predict() gives you values that are qualitatively similar to y (your labels). But what I'm seeking some clarity over is why are there only a few classes that have BOTH of these methods, e.g KMeans, PLSRegression, etc.

Why doesn't it make sense to either put both methods in every class, or never let these two occur in the same class together? For example, if KMeans needed to have a method that returns the Euclidean points, why not let it have a separate method? Implementing transform() to achieve this functionality, in my opinion, takes away the clear distinction between the two methods. Similarly, in PLSRegression, I haven't been able to understand the difference between the two methods.


texts: Purpose of visualizing high dimensional data?

texts:
There are many techniques for visualizing high dimension datasets, such as T-SNE, isomap, PCA, supervised PCA, etc. And we go through the motions of projecting the data down to a 2D or 3D space, so we have a "pretty pictures". Some of these embedding (manifold learning) methods are described here.

But is this "pretty picture" actually meaningful? What possible insights can someone grab by trying to visualize this embedded space?

I ask because the projection down to this embedded space is usually meaningless. For example, if you project your data down to principal components generated by PCA, those principal components (eiganvectors) don't correspond to features in the dataset; they're their own feature space.

Similarly, t-SNE projects your data down to a space, where items are near each other if they minimize some KL divergence. This isn't the original feature space anymore. (Correct me if I'm wrong, but I don't even think there is a large effort by the ML community to use t-SNE to aid classification; that's a different problem than data visualization though.)

I'm just very largely confused why people make such a big deal about some of these visualizations.


texts: How does time-series classification work?

texts:
So I have the multiple time series data in this format --

id  Week    X1    X2   X3   y

1   201901  10.   34.9 52.8 0

1   201902  20.   44.1 12.6 1

1   201903  30.   54.2 84.3 1

1   201904  40.   64.4 77.1 0

2   201901  10.   64.9 62.8 1

2   201902  20.   54.1 22.6 1

2   201903  30.   34.2 54.3 1

2   201904  40.   44.4 17.1 0

I want to predict y for the next two weeks. How can I approach this problem?

One way I thought is to train on the data set and then the test data would look like this -

id  Week   X1   X2    X3
1   201905 20.  45.2. 33.2
2   201905 10.  65.2. 23.2

Is this the correct way to do such a classification? Or should I train the model for each time series independently.

Thanks!


texts:
I have setup a four layer CNN designed to predict two classes. The two classes are more or less in the same ratio. The negative class is 55% of the data and the positive class makes up the remaining. When I train the network on the data and look at the final results, I see two problems:

The validation accuracy fluctuates. In fact, it can be thought of as oscillating. The validation loss increases and the accuracy however keeps hovering.
The final results indicate a possible reason for this behaviour. The negative classes are very well classified. However, the postive classes are very poorly classified.

My question is why is the network exhbiting this behaviour. I've enough training data, ~10k images in training, and the classes are more or less balanced (5400 in the negative and 4500 in the positive). The issue raised in point 2 increases with an increase in the number of epochs. I've tried decreasing the learning rate, different optimizers, increased the number of layers, decreased the number of layers, used transfer learning (VGG-16). I've also tried reducing the number of epochs, but this leads to poor classification in the negative class as well.


texts:
I have a daily series from Google Trends, using the range "today 3-m", but it comes that the last day is not available from this query. For example, today is March,24th and the last day using this query is March, 22th and I expected that it would be March, 23th.
If I take the series using the range "now 7-d", the day comes hourly and there is March, 23th. I would like to aggregate it and put it in the same measure as the series I obtained before. For this purpose, I need to know how the series is normalized. I understood that each time I take a series (one region and one word), it simple divided each point by the largest number in the range and multiply it by 100. Hence, the maximum point of the series is 100. Using this hypothesis, if I sum all the indexes in the second series by day, each day has the same denominator (but different from the first series). But in this case, the growth between adjacents days it has to be the same in the two series. But it is not what happened. So, I did not understand the normalization.
Could anyone help me, please?


texts: LSTM neural network for music generation

texts:
I am doing a project where I have to train a LSTM neural network to generate music. The first step is of course training it. I am using Keras on top of Theano for this task and music21 for feature extraction from MIDI files.
So far the data I extract from MIDI files has the following format:

[[69, -7, 1, 0.75],
 [62, -1, 0, 0.25] ... ]

These are the 4 features that I take from the files. They are then placed in a 3D numpy array with the shape - (number of notes in sequence, 1 (time steps per sample), 4 (features) ).
I create another array with the expected output for each time step being the data in the next time step. I've built the following model: http://pastebin.com/AHCJj1wa

The problem is that when it starts training the model I get some nonsensical output. First the epochs' output looks like this:

Epoch 1/100
0s - loss: 15.2986 - acc: 0.9738
Epoch 2/100
0s - loss: -3.3534e+02 - acc: 0.8265
Epoch 3/100
0s - loss: -3.8183e+02 - acc: 0.8265
Epoch 4/100
0s - loss: -3.8218e+02 - acc: 0.8265
Epoch 5/100
0s - loss: -3.8256e+02 - acc: 0.8265

It seems to be taking 0s per epoch and the loss just becomes larger and larger or stops changing at all (differs between runs).
Also at the end of the training cycle when I call .predict it prints something like that:

[[ 0.11846249 -0.13457553  0.08614471 -0.1007532 ]
 [ 0.14596225 -0.17533173  0.10007301 -0.08799653]
 [ 0.15234099 -0.19707087  0.10395571 -0.08419057]
 [ 0.14596225 -0.17533173  0.10007301 -0.08799653]
 [ 0.14340727 -0.1664145   0.0987886  -0.09037785]
 [ 0.14596225 -0.17533173  0.10007301 -0.08799653]
 [ 0.14340727 -0.1664145   0.0987886  -0.09037785]
 [ 0.14802463 -0.17770432  0.10296948 -0.08740482]
 ..... ]

I think that there is something fundamentally wrong with my model and/or representation of the data. Any ideas on what I can do or at least some pointers to where I can read about this?


texts: Action Recognition for multiple objects and localization

texts:
I want to ask questions regarding the action detection on the video with proposed frames. I've used Temporal 3D ConvNet for the action recognition on video. Successfully trained it and can recognize action on videos.
When I do inference, I just collect 20 frames from video, feed it to the model and it gives me the result. The point is that events on different videos are not similar in size. Some of them cover 90% of the frame, but some May 10%. Let's take as an example that two objects collided and it can happen on a different scale, and I want to detect this action.
How provide to the model exact position for the action recognition, if it can happen on a different scale with different objects? What comes to mind is to use Yolo to collect Regions of Interest and feed collected frames every time the 3D convnet. But if there are a lot of objects, the speed will be very slow. How to handle it?
Are there any end-to-end solutions for the action recognition with the object location proposal for the action recognition network?
I've already looked at papers and blogs, what people suggest, couldn't find the solution for the localization issues, so the action recognition model got the correct frames.


texts: NAB benchmark (Numenta Anomaly Benchmark)

texts: I have doubt on how to use NAB dataset for real time anomaly detection. The available datasets, for example the New York City taxi dataset, contains only 2 columns of timestamp and value, but where are the labels to predict the AUC for the dataset?


texts: Context Based Embeddings vs character based embeddings vs word based embeddings

texts:
I am working on a problem that uses English alphabets in the text but the language is not English. Its a mixture of English and different language text. But all words are written using English alphabets. Now, word-based pre-trained embedding models will not work here as it gives a random embedding to out of vocabulary words.
Now my question is that how the Context-based pre-trained embeddings deal with "out of vocabulary" words?
Besides, what's the difference between context-based embeddings and character-based embeddings?


texts: Best way to build a wave classification system

texts:
I want to make a classifier for waves such as following:

Above image is from: http://www.invisiblesbook.com/equal-temperament-tuning/

I believe, I will have to extract features from raw input using FFT or some other method and subject that to a CNN. What is the best way to go about building classifier for such waves? I am not particularly asking about neural-networks; any other method which is accurate and fast will also be of help.


texts: Histogram with financial (decimal) amounts vs. normal numeric

texts:
Take the following historgram data:

This is an item of "bin size" 1 from 0 onwards. However, I do not think this looks appropriate, as every time I have seen a histogram (or someone has requested it), it has unambiguous values, such as:

$ 0.00 - $0.99
$ 1.00 - $1.99
etc.

However, not even Excel does this correctly, so I was wondering if there was something like a suggested "significant figures" to apply to a histogram so that:

(1) It looks human, common-sensical, and proper; and
(2) There are not overlapping intervals [0, 1), [1, 2), etc.

Perhaps this is correct in a mathematical sense, but I'm looking for a more non-analytical intuitive representation of the pricing data. What is the suggested practice here?

P.S. Is this the correct site for this question? Or should it be on Cross Validated or StackOverflow?


texts: How to transform entire pandas data frame in one hot representation?

texts: I want all the columns one hot encoded without the need of listing out the columns or apply one hot encode one by one. I know how to do it one column then another.


texts: Graduate Degree Choices for Data Science

texts:
I'm currently finishing up a B.S. in mathematics and would like to attend graduate school (a master's degree for starters, with the possibility of a subsequent Ph.D.) with an eye toward entering the field of data science. I'm also particularly interested in machine learning.

What are the graduate degree choices that would get me to where I want to go?

Is there a consensus as to whether a graduate degree in applied mathematics, statistics, or computer science would put me in a better position to enter the field of data science?

Thank you all for the help, this is a big choice for me and any input is very much appreciated. Typically I ask my questions on Mathematics Stack Exchange, but I thought asking here would give me a broader and better rounded perspective.


texts: What This Systolic Blood Pressure Wants to Explain To Us?

texts: Good day. I know how thin plate spline is used for smoothing splines since I've been using it personally in my research field. I am just totally confused why lattice of knots (red and blue) are used in this figure with a contour plot. What this image really wants to expound to us? Any ideas folks?


texts: Does converting continuous variable to discrete(categorical) variable increases accuracy of a tree based model?

texts:
I've read other questions regarding if a continuous feature should be converted to categorical or not. But I'm interested in case of tree based classifiers such as Decision Tree, Random Forest, Gradient Boosted etc.

My intuition is that since tree based classifiers try to find the optimal split or the best test at each node, providing a categorical feature would make the splits more accurate than a providing a continuous feature.

My question is, doing the aforementioned pre-processing of data will lead to high accuracy in case of tree based models or the opposite? or it depends on the data?


texts: My data set is of shape (60,784,1000) with mini batches for input and (60,10,1000) for labels, should I shuffle only the 60 mini batches or the training examples themselves?


texts: Algorithm to apply Lasso and Ridge in Gradient descent

texts:
This my Gradient descent function -

Take ϴ as 0,0
Till n iterations
    hypothesis = Dot product of ϴ and feature vector X
    loss = hypothesis - target vector
    descent = Dot product of Transpose of X and loss divided by the size of X
    ϴ = theta - alpha*grad

Now how to modify this to include Lasso and Ridge regression?


texts: Surprise Library - RSME coming back as NaN

texts:
I'm using the Surprise library to train an SVD model based on some data I've collected from Steam. However, the RSME comes back as NaN when I try to train and test the model. For context, I'm using the z_score of the playtime that each person has for a particular game as their 'rating'. I'm new to this, so I don't know what's going wrong. I have snippets from my code shown below
Edit: I just found that my data is heavily skewed, so I have ratings way above 3. I imagine this might be causing a problem. If you all see anything else that might be causing an issue let me know!
The dataframe I'm working off of. user_id and game_id are strings, z_score is a float

The code I used


texts: I have made a model that takes vacancy data and classifies it with (bi)LSTM. The variables of the initial dataset are: positiontitle vacancytext and of course the label. The model is successful in classifying the variable vacancytext. At this point I am trying to also include the variable positiontitle making it a multivariate text classification problem. However, whenever I look up information about multivariate (bi)LSTM text classification it only gives me results for multivariate time series forecasting. Anyone has an idea how can I find information about multivariate text classification with (bi)LSTM, so I can use both the variablespositiontitle and vacancytext for my model?


texts: Why the loss is nan by using linear activation function in the last layer?

texts:
I want to use neural network to solve a simple regression problem, and I try to program by myself accroding to lecture Backpropagation and Neural Networks .
However, I meet loss divergence problem.

My neural network can be descried as:

$l_{1}=\frac{1}{1+e^{-(W_0 x + b_0)}}$

$l_{2}={W_1 l_1 + b_1}$

And loss is:

$loss = {(y-l_2)^T(y - l_2)}$

import numpy as np
import matplotlib.pyplot as plt

np.random.seed(1)
x_pre = 2*np.random.normal(size = (1000,1))
y = x_pre**2 + -0.2*np.cos(3*np.pi*x_pre)
y = y.reshape(y.shape[0],1)

def standard(data):
    mu = np.mean(data)
    std = np.var(data)
    return (data - mu)/std
x = standard(x_pre)

ndim = 10
w0 = 2*np.random.random((1,ndim))-1
w1 = 2*np.random.random((ndim,1))-1
np.random.seed(10)
b0 = np.random.normal(size = (1000,ndim))
b1 = np.random.normal(size = (1000,1))
lr = 1
for j in range(4):
    l1 = 1/(1+ np.exp(-(np.dot(x,w0)+ b0)))
    l2 = np.dot(l1,w1)+ b1#1/(1+ np.exp(-(np.dot(l1,w1)+ b1)))
    l2_delta= np.mean(y - l2)*2*(y-l2)#mse
    l1_delta = l2_delta.dot(w1.T)*(l1*(1-l1))
    w1 += lr*l1.T.dot(l2_delta)
    w0 += lr*x.T.dot(l1_delta)
    b0 += lr* l1_delta
    b1 += lr*l2_delta
    print('loss',np.mean(y - l2))
l1 = 1/(1+ np.exp(-(np.dot(x,w0))))
l2 = 1/(1+ np.exp(-np.dot(l1,w1)))
y_hap =  l2

print(np.sum(np.square(y-y_hap)))
plt.plot(np.arange(len(y)),y,'r')
plt.plot(np.arange(len(y_hap)),y_hap,'g')
plt.show()

output result:

loss 3.485080512494127
loss -30525.316587393125
loss -3293457250652.145
loss -5.777133209515429e+28

Anyone knows how to solve it?


texts: Measuring performance of different classifiers with different sample sizes

texts:
I'm currently using several different classifiers on various entities extracted from text, and using precision/recall as a summary of how well each separate classifier performs across a given dataset.

I'm wondering if there's a meaningful way of comparing the performance of these classifiers in a similar way, but which also takes into account the total numbers of each entity in the test data that's being classified?

Currently, I'm using precision/recall as a measure of performance, so might have something like:

                    Precision Recall
Person classifier   65%       40%
Company classifier  98%       90%
Cheese classifier   10%       50%
Egg classifier      100%      100%

However, the dataset I'm running these on might contain 100k people, 5k companies, 500 cheeses, and 1 egg.

So is there a summary statistic I can add to the above table which also takes into account the total number of each item? Or is there some way of measuring the fact that e.g. 100% prec/rec on the Egg classifier might not be meaningful with only 1 data item?

Let's say we had hundreds of such classifiers, I guess I'm looking for a good way to answer questions like "Which classifiers are underperforming? Which classifiers lack sufficient test data to tell whether they're underperforming?".


texts:
I have some data in following format:

kid1, 6am, sleeping
kid1, 7am, sleeping
kid1, 8am, sleeping
kid1, 9am, playing
kid1, 10am, playing
kid1, 11am, eating
kid2, 6am, sleeping
kid2, 7am, sleeping
kid2, 8am, sleeping
kid2, 9am, playing
kid2, 10am, playing
kid2, 11am, eating

We can easily spot the pattern here that kid1 slept for 3 hours, played for 2 hours etc.

Same thing I want to do using machine learning. Some people suggested using decision trees, graphs etc etc. However, it is not a labeled data and the activities like sleeping, eating can also be in some good number. Thought of unsupervised clustering and time series too, but couldn't get it working.

Can someone suggest an actual working example (if some similar code is there, I can tweak).

I believe this is an old problem of data science, just a right hint if someone can give, that would be helpful.


texts: I know we need to test our model on onseen data, but isn't that test set are for? Also what will happen if we increase K value in kfold?


texts: best python library for finding sequential rules mining?

texts: I want to do sequence learning for that I want to find frequent sequential rules ,this rule consider the order of occurrence


texts: How to expand abbrevations in text during preprocessing?

texts:
Im doing preprocessing on english text data. I have some domain specific abbreviations, for which i'm maintaining internal dictionary with key-value pairs. The problem i'm facing is the text has abbreviations in plural forms with and without contractions like:

Mgr's = manager
mgrs = manager
mgr = manager

All 3 points to a manager. Im able to capture the plural form with contractions using a regex(r"'s") and removing the 's' but, in case of no contractions i'm creating one more entry in the dictionary with plural form of the abbreviations.
Im somehow feel this is duplication and not a clean approach. Is there any better solution to address this problem?
Any immediate help on this is much appreciated. Thank you


texts:
I am thinking of two use cases:

You submit a resume in PDF format to a web site and it extracts your contact information, job titles, etc.
You receive an email from a friend that says, "let's have lunch next Tuesday" and your email program detects it and asks if you want to save a new calendar entry for "lunch on Tuesday".

I can see how this can be done with both ML and with (very sophisticated) regex. What approaches are typically used for these scenarios? I'm assuming that the same approach can be applied to both of the scenarios above?


texts: Clustering sequences of sentence embeddings

texts:
I have a sequence of events, right now I am not worried about their actual times, just the order. This is a sequence of web page views.

I have modelled my data as the following, where each element represents the category of the web page.

user_sequence = ['A', 'A', 'B', 'C', ...]

Following this I used the code and approach from this paper: Sequence Graph Transform

My question is how could I represent more complex data in my sequences, for example, we have an embedding representing the page content, along with features including the dwell of the user on each page.

So to summarise, the goal is to do process sequences of rich event data in an unsupervised manner.


texts: images from training set are different from images of test set

texts:
I am doing image classification with CNN and I have a training set and a test set with different distributions. To try to overcome this problem I am thinking about doing a standardization using Imagegenerator, but I am encoutering some problems. Here is the part of the code I am working on:

trainingset = '/content/drive/My Drive/Colab Notebooks/Train'
testset = '/content/drive/My Drive/Colab Notebooks/Test'

batch_size = 32
train_datagen = ImageDataGenerator(
    featurewise_center=True,
    featurewise_std_normalization=True,
    rescale = 1. / 255,\
    zoom_range=0.1,\
    rotation_range=10,\
    width_shift_range=0.1,\
    height_shift_range=0.1,\
    horizontal_flip=True,\
    vertical_flip=False)

train_datagen.fit(trainingset);

train_generator = train_datagen.flow_from_directory(
    directory=trainingset,
    #target_size=(256, 256),
    color_mode="rgb",
    batch_size=batch_size,
    class_mode="categorical",
    shuffle=True
)

test_datagen = ImageDataGenerator(
    featurewise_center=True,
    featurewise_std_normalization=True,
    rescale = 1. / 255)

test_datagen.fit(testset);

test_generator = test_datagen.flow_from_directory(
    directory=testset,
    #target_size=(256, 256),
    color_mode="rgb",
    batch_size=batch_size,
    class_mode="categorical",
    shuffle=False
)

num_samples = train_generator.n
num_classes = train_generator.num_classes
input_shape = train_generator.image_shape

classnames = [k for k,v in train_generator.class_indices.items()]

print("Image input %s" %str(input_shape))
print("Classes: %r" %classnames)

print('Loaded %d training samples from %d classes.' %
(num_samples,num_classes))
print('Loaded %d test samples from %d classes.' %
(test_generator.n,test_generator.num_classes))

so, what I am trying to do is using in the Imag  genereator the fields featurewise_center=True and featurewise_std_normalization=True to do standardization, but if I try to fit the generator to the trainingset by doing train_datagen.fit(trainingset); I get the following error:

    ValueError                                Traceback (most recent call last)
<ipython-input-16-28e4ebb819be> in <module>()
     23     vertical_flip=False)
     24
---> 25 train_datagen.fit(trainingset);
     26
     27 train_generator = train_datagen.flow_from_directory(

1 frames
/usr/local/lib/python3.6/dist-packages/numpy/core/_asarray.py in asarray(a,
dtype, order)
     83
     84     """
---> 85     return array(a, dtype, copy=False, order=order)
     86
     87

ValueError: could not convert string to float: '/content/drive/My Drive/Colab
Notebooks/Train'

Can somebody please help me? Thanks in advance.

[EDIT] I am trying to adapt what is written here to my problem.

[EDIT_2] I think the problem is that .fit() takes as parameter a numpy array, while I am trying to pass to it a string, which is the path for the images.

But I don't understand now how to do, because I should transform this to a numpy array in order to do the fit.


texts: I want to visualize statistics from two large dataframes with similar data in two pie charts. I would like for these two charts to use the same color when the same category appears in the chart. I have found similar questions online but all the advice include manually creating a dictionary of categories, which seems to be too complicated because of the large number of categories I have in the dataset.


texts: What is the best way to treat datetime in the preprocessing step of machine learning

texts:
I have two datetime columns in my dataset. What I have done so far I have extracted year, month, dayofweek and hourofday from these columns.
So as you expect they will be something like this:
2015-5-5 08:21:20   ----> 2015   5   5   8

So my question is that what is the best way to normalize these numbers. Because I think that year or other numbers will dominate my machine learning model.
I have not found any article regarding this, all explaining till this point that we convert them to year month ...
Thanks.


texts: How to model to predict hotel booking abnomality?

texts:
We are trying to build a model, gathering specific hotels booking data, try to find the pattern how the hotel is booked, stayed, what type of people lived, how many bookings average per day. The bookings might vary from weekdays to weekend, also from winter to summer, normal days and vacation period. All these factors are accountable.

Then, as the time passes by, we want to know if the booking becomes abnormal, e.g. normally, young couple books the hotel quite a lot, all the sudden, a bunch of business men checked in for a couple of days.

Since in this case, we don't have samples/labels for normality and abnormality, we started thinking use unsupervised learning, like clustering for a start. Say, we construct a sample(booking features for every week) going back all the way to the beginning of the year. And then, we try to cluster them, then, every week, we calculate the current week, and see if it belongs to any clusters? or it is an abnormal point stands out required attention.

Is this a reasonable approach or there are some better ways?


texts: How to calculate the Probability for the Unconditional Node in the Bayesian Belief Network?

texts:
In the popular example for Bayesian Belief network Burglary Alarm how is the probability for burglary P(B) and earthquake P(E) calculated as 0.001 and 0.002 respectively?

Is it an assumption made or there is some calculation involved ? I can understand the conditional probabilities for the child nodes but not sure how the probability for the nodes Burglary and Earthquake are getting calculated?


texts: Dropout on inputs instead on outputs = DropConnect?

texts:
Is dropping out parts of the Input vector better than dropping out parts of the Output vector?

The latter literally makes this same neuron invisible to any further layers. On the contrary, ignoring pieces of the input means some of the further neurons will be able to see this neuron.

Is masking the input pretty much the same as masking the weights (aka DropConnect) and thus gives a higher quality regularization?


texts: Finding dominating attributes with in the clusters generated

texts:
I am having a dataset of customers where each customer is represented as some feature vector and I am applying K-means algorithm to this dataset. On the basis of those features, I can abstract and give names to these clusters. But I want to map demographic features on these clusters to validate my clusters whether they make sense or not.

Now for example I have original data set of 100 customers with 90 females and 10 males.

I suppose find 4 clusters and each is of size 25. In Cluster No: 1,2 and 3, I have all males and no females. In Cluster No: 4, I have 15 males and 10 females.

How to map genders to these clusters to describe clusters as male or female dominating ?

Am I making sense here ?


texts:
I am predicting the number of cars from a traffic dataset.
Here is my data dictionary:
The ‘Traffic-Major-Roads(kilometres)’ file contains the following variables (variable names
are in bold):

Year - Traffic volumes are shown for each year from 2000 onwards.
CP (count point) – a unique reference for the road link that links the AADFs to the road
network.
ONS GOR Name – the former Government Office Region that the CP sits within.
ONS LA Name – the local authority that the CP sits within.
Road – this is the road name (for instance M25 or A3).
RCat – the classification of the road type (see data definitions for the full list).
S Ref E – Easting coordinates of the CP location.
S Ref N – Easting coordinates of the CP location.
A-Junction – The road name of the start junction of the link
B-Junction – The road name of the end junction of the link
LenNet – Total length of the network road link for that CP (in kilometres).
PC – Traffic volume (in thousands of vehicle kilometres) for pedal cycles.
2WMV – Traffic volume (in thousands of vehicle kilometres) for two-wheeled motor
vehicles.
Car - Traffic volume (in thousands of vehicle kilometres) for Cars and Taxis.
Bus – Traffic volume (in thousands of vehicle kilometres) for Buses and Coaches
LGV – Traffic volume (in thousands of vehicle kilometres) for LGVs.
HGVR2 – Traffic volume (in thousands of vehicle kilometres) for two-rigid axle HGVs.
HGVR3 – Traffic volume (in thousands of vehicle kilometres) for three-rigid axle HGVs.
HGVR4 – Traffic volume (in thousands of vehicle kilometres) for four or more rigid axle
HGVs.
HGVA3 – Traffic volume (in thousands of vehicle kilometres) for three or fourarticulated axle HGVs.
HGVA5 – Traffic volume (in thousands of vehicle kilometres) for five-articulated axle
HGVs.
HGVA6 – Traffic volume (in thousands of vehicle kilometres) for six-articulated axle
HGVs.
HGV – Traffic volume (in thousands of vehicle kilometres) for all HGVs.
AMV – Traffic volume (in thousands of vehicle kilometres) for all motor vehicles.

I need to predict the variable AMV.
So, I have one-hot encoded Road, and kept date, time in my features.
But, number of Roads being very large. I have too many features.
Can you please suggest how should I proceed?


texts: plotting time series data using matplotlib python

texts:
I am trying to visualize Time series data is as follows
following is my code to plot the data
plt.plot(data['date'], data['c_16_avg_a'])
plt.xticks(rotation='vertical')
the date is getting truncated here. and the plot looks wired.
Any suggestions will greatly be appreciated


texts:
from sklearn.model_selection import RandomizedSearchCV,GridSearchCV
import xgboost

classifier=xgboost.XGBClassifier()
random_search=RandomizedSearchCV(classifier,param_distributions=params,n_iter=5,scoring='roc_auc',n_jobs=-1,cv=5,verbose=3)
random_search.best_estimator_

  .AttributeError: 'RandomizedSearchCV' object has no attribute 'best_estimator_'


texts: Combine RepeatedStratifiedKFold and crossval

texts:
Is it okay to combine RepeatedStratifiedKFold and cross_val_score?

The result in the example below are 30 accuracy values (3x10 splits).
How to calculate the final accuracy value for the 3 times 10 fold cv?

cv = RepeatedStratifiedKFold(n_splits=10, n_repeats=3, random_state=rand)
rf = RandomForestClassifier(n_estimators=e, random_state=rand)
scores = cross_val_score(rf, data, Y, scoring='accuracy', cv=cv, n_jobs=-1)


texts: Tensorflow (or Keras) vs. Pytorch vs. some other ML library for implementing a CNN

texts:
I am looking into implementing a convolutional neural network for a research problem. I've heard of deep learning libraries like Pytorch and Tensorflow and was hoping to get some additional information about their suitability for my needs.

I haven't looked much into Pytorch, and have only briefly read about Tensorflow. I don't hear very nice things about Tensorflow in terms of ease of use. I hear Pytorch is easier to use. But there seems to be more tutorials for Tensorflow, and specifically for creating CNNs.

What sort of questions should I be asking myself in determining which library would best suit my needs?


texts: Plotting Loss on Validation Set for Text Summarization Neural Net

texts:
I am using a seq2seq encoder-decoder neural net with attention to summarize text, but I think it may be overfitting. https://github.com/dongjun-Lee/text-summarization-tensorflow

Is there a way to plot loss vs num_epochs for both the training and validation set? Is this an interpretable measure?

From my understanding ROUGE score is necessary because multiple different summaries can be considered a correct answer, so does that mean you cannot use loss on the validation set vs num_epochs as a measure for over/underfitting? Are there any other ways to check for over/under fitting?


texts:
I was wondering why there isn't a feature built into common-use ML libraries, like Keras, that plugs many different combinations of layers and nodes to multiple models and trains them simultaneously to single out the best NN architecture for your problem?

For example, given training data, validation data, and a loss function, it compares a model consisting of two hidden Dense layers with 256 neurons each, and another model consisting of two hidden Dense layers, the first with 256 and the second with 64. It would then save the model with higher accuracy according to the input loss function.

Does something like this exist already? I know something like this exists with SKLearn's GridSeachCV, but I wasn't sure if that's a common practice outside of SKLearn. I feel like I might be oversimplifying a complex problem.

Thank you!


texts:
I am currently looking at a paper by Hubara et al  on binarized neural network.

I am stuck in understanding algorithm 2 of the paper. The algorithm uses shift-based (bit-shifting) AdaMax, where AdaMax is an extension of the Adam optimizer. In particular, they are using

$$m_{t} = \beta_{1}.m_{t-1} + (1 - \beta_{1}).g_{t} $$

$$v_{t} = \max(\beta_{2}.v_{t-1}, |g_{t}|) $$

$$\theta_{t} = \theta_{t−1} − (\alpha\oslash(1−β_{1}^t)).(m_{t} \oslash v_{t}^{-1} ) $$

where $g_{t}$ is the gradient, $\theta_{t-1}$ is the previous parameter, $\alpha$, $\beta_{1}$, $\beta_{2}$ are learning rate, and the betas of Adam optimizer. They stated that $\oslash$ stands for both left and right  bit shift. I know the left shift and the right shift on its own, but I am not sure how do we have both at the same time? Help will be much appreciated. Thank you.


texts: Linear regression using math or machine learning? Why even use machine learning for this?

texts: I have studied statistical math and is now taking a course in machine learning. The first example the teacher talked about is how to find a linear trend line using machine learning. Why would anybody do that? In my statistical courses we calculated linear trend-lines on datasets using far less computer power than a machine learning approach would use according to my understanding. Is linear regression using machine learning only a nice example to learn machine learning basics, or are there any real world reasons for why anyone would use machine learning to find such a simple trend line?


texts: Is Nvidia Jetson product family also suitable for machine learing model training?

texts:
I recently came accross these products (Nvidia Jetson) and they are all tagged as "edge", so i think they are designed only for machine learning inference and not model training.
They are quite interesting for their low power consumpion and price (eg: Jetson Nano) so i hope they are suitable also for model trainig.
So i would ask if someone may clarify this aspect about the focus of the product.


texts:
I am using inertial sensors to capture motion from the wrist/ ankles of subjects and synthesising this data into a feature set to be able to classify the activity performed by the subject (i.e. standing, walking, sitting).

On top of the metrics output by the sensors I would also like to be able to use the orientation of the limb segments for classification. I have been using an existing algorithm to determine the quaternions and can also convert these into Euler angles if need be.

My question is how best process these quaternions in order to feed them into a classifier to provide information about limb segment orientation to the classifier. The options for feeding the data into the classifier as far as I'm aware are as follows:

1) Feed each element of the quaterion as a feature: My concern is that each element of the quaternion is clearly non-linear and the classifier I use may not be sophisticated enough to be able to determine useful orientation information from this metric

2) Convert to Euler angles: This again suffers from a non-linear relationship with orientation relative to a world space depending on order of successive rotations (may be solved using extrinsic euler angles?)

3) Calculate Quaternion modulus from the non-unit quaternion: I have seen this method suggested in a paper on classification of EEG signals but I'm not sure of it's usefulness for this application. To be honest I'm not entirely sure on the what Quaternion modulus actually represents in real term other than for it's application in normalising quaternions.
https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4813911/#B40-sensors-16-00336

Many thanks in advance for any help!


texts: I'm wondering if there has been some work done about using autoencoder versus using word2vec to produce word embeddings. Autoencoder could learn to map contexts words with themselves while word2vec usually maps one word with its context or a context with its word. Did you see some work comparing these two approaches ?


texts:
Currently, I am working on my thesis which is built on LSTM networks and I am using PyTorch library. However I am struggling to solve the conceptual problem of archiving trained models.

To make the question more clear; I am saving models in a archicture that I can give this form /models/model-with-loss-2.634221 as an example. But with this form, it is hard to determine which is which. I tried use more detailed form like 1-layered-100-epoch-128-batchsize-...-etc, but it is also hard to read and determine.

What is your way that you think is most productive to handle such operation?

By the way I am not sure this is the correct network ask this question on, you can drop an comment if it is not.


texts: Evaluate a model based on precision for multi class classification

texts: I have a model that predicts the level of injury over 3 classes: Low, Medium and High. I wish to optimize the model parameters on the scoring basis of precision. However, precision is class specific, we can determine the precision of low, medium and high separately. Is there a way to determine something like "Overall Precision" from the confusion matrix?


texts:
I want to use Deep Q-learning (specifically DDQL by Hasselt et al. 2015, but it is the same principle) in a non-episodic task (continuing). I know that it is possible to use Q-learning in continuing tasks; however, I'm not sure if I should modify my model.
Until now, I've found little information, and the closest article that I have read is this website, where the use of mean reward is explained based on Sutton's book. The explanation makes sense to me, but I'm not completely sure, and I'm relatively new on the field.
I've grateful if anyone could give me some insights on using DQL in non-episodic tasks or a reference where I can look it. I especially care about the computation of the update.
Thanks in advance,
Tomás


texts: Are there established good algorithms for incremental feature learning for a neural network? Do any python ML libraries implement such algorithms?

texts:
I'm working with a high-dimensional dataset, and have found that my attempts at dimensionality-reduction hurt the accuracy of downstream classifiers, suggesting that I am effectively losing information in the course of feature-compression.

Furthermore, when I sort the features by importance (per estimation via random forest) and then train neural networks on different numbers of the most important features, I find that the resulting classifier accuracy stops increasing before I have exhausted all of the useful-seeming features.  I suspect that simultaneous training on many thousands of features is the problem, and I'd like to simplify the problem by learning from the features incrementally, that is:

I'd like to explore training a neural network adding a few features at a time: e.g. train using the top 100 features, then add in the next 100, train again (using coefs from previous round plus random coefs for the new part of the input layer), rinse, repeat.

NOTE that this "incrementalness" is across features, not samples; it doesn't affect whether training is incremental ("online") with respect to samples.

I gather this sort of thing has been tried before with some success but I haven't been able to find implementations in the Python ML libraries I've checked, nor can I find good explanations of the details of successful algorithms.

I have about 70K examples, divided over 6 balanced classes. For most of my work I care only about distinctions between a total of 4 classes, so I merge two pairs of the initial classes and end up with a total of 4 (I end up with unbalanced training sets as a result, but since the two larger classes are the ones whose classification is most important, I'm fine with that for now). I have a total of about 28K features, but am finding that I get best classifier performance when using no more than the top 4K features (pre-sorted by importance via Random Forest).

Does my approach seem reasonable?
Are there accepted published detailed algorithms (e.g., is it reasonable to add the features as extensions of the original input layer, or is another approach preferred)?
Do any Python libs, particularly any compatible with scikit-learn, implement such an algorithm?